a2) United States Patent
`US 6,219,032 B1
`(10) Patent No.:
`Rosenberg etal.
`(45) Date of Patent:
`*Apr. 17, 2001
`
`
`US006219032B1
`
`(54) METHOD FOR PROVIDING FORCE
`FEEDBACKTO A USER OF AN INTERFACE
`DEVICE BASED ON INTERACTIONS OF A
`CONTROLLED CURSOR WITH GRAPHICAL
`ELEMENTSIN A GRAPHICAL USER
`INTERFACE
`.
`Inventors: Louis B. Rosenberg, Pleasanton, CA
`(US); Scott B. Brave, Cambridge, MA
`(US)
`
`(75)
`
`(73) Assignee:
`
`Immersion Corporation, San Jose, CA
`(US)
`
`(*) Notice:
`
`This patent issued on a continued pros-
`ecution application filed under 37 CFR
`1.53(d), andis subject to the twenty year
`patent
`term provisions of 35 U.S.C.
`154(a)(2).
`.
`Lo.
`.
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 08/571,606
`.
`Filed:
`
`Dec. 13, 1995
`
`(22)
`
`er
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 08/566,282,filed on
`Dec.
`1. 1995. now Pat. No. 5.734.373.
`,
`,
`7?
`Int. C1? oe G09G 5/08; A63F 13/00
`(51)
`(52) US. Ch ce 345/157; 345/145; 463/37
`(58) Field of Search 0...ees 345/145, 146,
`345/156, 157, 158, 161, 163; 395/326,
`339, 340, 348, 349, 352, 353, 99; 463/30,
`31, 36, 37, 38; 318/568.11, 568.25; 364/578;
`340/407.1, 407.2; 74/471; 273/148 B; 434/45;
`244/223
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`3,490,059
`3,531,868
`
`1/1970 Paulsen et al. oo. eee 73/133
`
`10/1970 Stevenson 0... eeeeeeeeeneeee 33/174
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`0265011 Al
`4/1988 (EP).
`0607580 Al
`7/1994 (EP).
`.
`.
`(List continued on next page.)
`OTHER PUBLICATIONS
`
`Adachi, Yoshitaka et al., “Sensory Evaluation of Virtual
`Haptic Push—Buttons,” Technical Research Center, Suzuki
`Motor Corporation, Nov. 1994.
`(List continued on next page.)
`.
`.
`Primary Examiner—Steven J. Saras
`.
`Assistant Examiner—PaulA.Bell
`(74) Attorney, Agent, or Firm—JamesR. Riegel
`(57)
`ABSTRACT
`
`A method and apparatus for providing force feedback to a
`user operating a human/computer interface device in con-
`junction with a graphical user interface (GUI) displayed by
`a host computer system. Aphysical object, such as a joystick
`or a mouse, controls a graphical object, such as a cursor,
`within the GUI. The GUI allowsthe user to interface with
`operating system functions implemented by the computer
`system. A signal is output from the host computer to the
`interface device to apply a force sensation to the physical
`object using one or more actuators. This desired force
`sensation is associated with at least one of the graphical
`objects and operating system functionsofthe graphical user
`interface and is determined bya location of the cursor in the
`GUI with respect to targets that are associated with the
`graphical objects. The graphical objects include icons,
`windows, pull-down menus and menu items, scroll bars
`(“sliders”), and buttons. The force sensation assists the user
`to select a desired operating system function or physically
`informsthe user of the graphical objects encountered bythe
`cursor within the GUI. A microprocessor local to the inter-
`face apparatus and separate from the host computer can be
`used to control forces on the physical object.
`
`2,906,179
`
`Q/IV9S9 BOWEL weescecccscesesssseesseteteseseeees 90/13.5
`
`58 Claims, 27 Drawing Sheets
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`4 p~580
`
`
`
`
`
`
`
`
`
`
`
`
`Lo
`
`
`
` Valve Exhibit 1050
`
`Valve Exhibit 1050
`Valve v. Immersion
`Valve v. Immersion
`
`

`

`US 6,219,032 B1
`
`Page 2
`
`T/A992 Thomas, Jr oe ceeeeseeee 341/20
`5,128,671
`U.S. PATENT DOCUMENTS
`
`7/1992 Marinaccio etal.
`». 433/72
`5,131,844
`_
`vo 340/710
`7/1992 Clark .....
`5,132,672
`pees
`vio87 pheldon Ct AL. one ane
`
`273/148 B
`8/1992 Openiano
`5,139,261
`053,
`7
`TWANO ooeeeeevnestereseeenseeteesees
`
`8/1992 Edwards wssssessssstnunnee 367/127
`5,142,506
`Tereon2
`toes Ferringtonebale sven SMP
`
`9/1992 Menahem seeeevsrererer TAIAT XY
`5,142,931
`3,800,958
`6/1975. Fister et al. ceccscssseennen 128/2 X
`
`
`..
`ve A14/5
`9/1992 Burdea et al.
`5,143,505
`3,944,798
`3/1976 Eaton sseesssesseniestsnsnssee 235/151.3
`
`9/1992 Hollis, Jr. et al.
`395/275
`5,146,566
`4,114,882
`9/1978 Mau ..
`273/85 G
`
`
`9/1992 McDonald...
`- 364/560
`5,148,377
`4,148,014
`4/1979 BUYSOM ceessssssescussessseeeenneeee 340/709
`
`1/1993 Culp beneeeeeeesec ene ceeeeneeessenres cesses 73/510
`5,178,012
`4,216,467
`8/1980 COISTON ceccccccseccsecseseseesceece 340/365 L
`
`
`T/1993 GLYMM wee cee steeeeeeeeee 364/566
`5,181,181
`4,398,889
`8/1983 Lam cecccccccccccssescsesscsessesesessesses 434/45
`1/1993 Lang veeeccccseessesecteeeceseees 341/20
`5,182,557
`4,448,083
`...
`. 73/862.04
`5/1984 Hayashi
`2/1993 Erdmanet al. oe 364/474.05
`5,184,306
`4,477,043
`10/1984 Repperger
`. 244/223
`
`
`2/1993 Kramer........
` w+. 364/806
`5,184,319
`4,477,973
`10/1984 Davies .....
`33/1 CC
`
`2/1993 Good et al. vee ereee 318/432
`5,185,561
`4,550,221
`10/1985 Mabusth...
`.. 178/18
`
`Z/L9DB ROWeN oe eeecececeeeeeceeeeeee 434/114
`5,186,629
`4,550,617
`11/1985 Fraignieretal.
`. 73/862.04
`. 33/502
`2/1993 Takahashietal.
`5,187,874
`4,571,834
`2/1986 Fraser et al.
`.
`.. 33/1 PT
`
`3/1993. McMurtryetal. .
`.. 33/503
`5,189,806
`4,593,470
`33/1 CC
`6/1986 Davies ..
`
`
`3/1993 Tokunaga.....
`..- 395/161
`5,195,179
`4,601,206
`7/1986 Watson.....
`... 73/514
`
`364/474.03
`4/1993 Fujimaki
`5,204,824
`4,632,341
`12/1986
`. 244/230
`
`.. 73/865.8
`5/1993 Baxter ...
`5,209,131
`4,679,331
`7/1987 Koontz ..
`... 33/551
`
`
`6/1993 Schuler .
`w. 318/561
`5,220,260
`4,688,983
`. 414/735
`8/1987 Lindbom
`
`6/1993 Radkeet al
`. 318/568.1
`5,223,776
`4,689,449
`8/1987 ROSEN vo.
`eeesseceeseeeeeeeeeeeeeeeee 200/6
`
`74/471 XY
`7/1993 Chuang.....
`5,228,356
`4,692,756
`9/1987 Clark .....
`. 340/709
`
`7/1993 Guthrieet al.
`. 433/72
`5,230,623
`4,703,443
`10/1987 Moriyasu .
`. 364/559
`
`
`8/1993 Culver oe.
`w FA/A71
`5,235,868
`4,704,909
`11/1987 Grahn et al. woe 73/862.04
`
`
`. 318/568.1
`9/1993 Kasagamiet al.
`5,243,266
`4,734,685
`3/1988 Watanabe .0... ee eteeeeee 340/710
`
`
`4,750,487 5,251,127=10/1993 Raab o.6/1988 Zanetti ..... 128/303 B 364/413.13
`
`
`
`
`9/1988 Trieb et al. oe 364/559
`wa. 364/559
`4,769,763
`5,251,156
`10/1993 Heier et al.
`..
`
`
`11/1988 Kley et al. veces 340/365
`11/1993 Chapmanetal.
`.. 33/502
`4,782,327
`5,259,120
`
`... 156/64
`......
`4,787,051
`11/1988 Olson....
`. 364/518
`5,259,894
`11/1993 Sampson.
`
`
`12/1988 Brummett occ enone 128/653
`. 341/20
`11/1993 Lowetal.
`4,791,934
`5,262,777
`
`. 318/561
`4,798,919
`1/1989 Miessler et al. oe eee 178/18
`5,264,768
`11/1993 Gregory
`«. 434/29
`1/1989 Cemenskaetal
`.. 60/393
`4,800,721
`5,275,565
`1/1994 Moncrief
`
`3/1994 Ledley 0...eee 345/161
`2/1989 Kendig et al. occ 318/648
`4,803,413
`5,296,846
`
`3/1989 [Tilton oe eeeeeeeteeeee 73/862.04
`4,811,608
`5,296,871
`B/1994 Paley weeccccsccsessecseesererensees 345/163
`4,819,195
`4/1989 Bell etal.
`364/571.05
`5,298,890
`3/1994 Kanamaru et al.
`345/157
`
`
`6/1989 LaBiche et al. oe. 364/709.11
`5/1994 Venolia et al. eee 345/163
`4,839,838
`5,313,230
`4,849,692
`10/1994 Dowet al. woe 128/662.06
`TIV989 Blood vost eeeeee esse eee es 324/208
`5,351,692
`4,868,549 10/1994 Burdea et al.oo...eee 414/59/1989 Affinito et al. . . 340/710 5,354,162
`
`
`
`
`
`
`
`11/1989) Duimel oe eee teeeeeeeeee 341/20
`10/1994 Anderson... ee eeeteeteeeeeee 345/166
`4,879,556
`5,355,148
`4,888,877
`12/1994 Gilligan et al. 345/157
`12/1989 Enderle et al. oo. eee 33/559
`5,374,942
`
`1/1990 ‘Tomelleri
`....
`... 33/503
`4,891,889
`5,379,663
`1/1995 Hara ......
`TA/ATL XY
`
`
`
`4,907,970 3/1990 Meenen, Jr.ceeeeeeeeeeeeee 434/45 5,384,460 1/1995 Tseng .... 250/231.14
`
`
`
`1/1995 Teig et ab.eee 395/161
`4,907,973
`B/1990 AON oo.
`eeeeeeeeeeeeeseeeteeeteeeteees 434/262
`5,386,507
`
`.... 340/709
`4,935,728
`6/1990
`5,389,865
`2/1995 Jacobusetal.
`318/568.11
`
`3/1995) Brimhall occ eens 345/161
`TIV9IO Sapia oeecccceceeeeseerees 364/571.01
`4,942,545
`5,396,266
`FIA99O Blood woe
`cneeeeeeee 324/207.17
`4,945,305
`5,396,267
`B/L99DS Bouton oc. eceeseeeeeeeeeeeee 345/168
`
`
`364/571.05
`4,945,501
`7/1990 Bell et al
`5,397,232
`3/1995 Tayloret al.
`606/130
`... 364/200
`10/1990 Gorniak ...
`3/1995 Hill ooee eseeeeeeeeeeteenees 345/145
`4,961,138
`5,398,044
`
`10/1990 Herzog.....
`4,961,267
`... 33/503
`5,402,582
`AM995 Raab oe eeeteeeereeeee 33/503
`
`
`.
`w. 33/502
`4,962,591
`10/1990 Zeller et al.
`5,405,152
`4/1995 Katanics et al
`273/438
`
`... 33/502
`S/1995 Raab wee ee ececsceeeseceeceeeeee 33/503
`4,982,504
`1/1991 Séderberg etal.
`5,412,880
`. 74/471
`4,982,618
`1/1991 Culver 0.
`5,414,337
`S/1995 Schuler... ceeeeeeeeerenenees 318/561
`
`
`. 178/18
`. 606/9.1
`4,983,786
`1/1991 Stevens et al.
`.
`5,417,696
`5/1995 Kashubaet al.
`.
`... 380/25
`6/1995 Davidson et al. we 395/275
`§,007,085
`4/1991 Greaniasetal.
`5,428,748
`4/1991 Siva wo
`TA/AT1L XY
`$007,300
`5,429,140
`FA995 Burdea et al. eee 128/774
`
`8/1991 McMurtryet al
`... 33/556
`7/1995 Lipson woe
`eecsseeeteeeeeeees 463/3
`5,040,306
`5,435,554 *
`
`
`9/1991 Behenskyetal. ..
`.. 434/45
`5,044,956
`5,436,542
`7/1995 Petelin et al.
`318/567
`
`
`9/1991 Watanabeetal.
`128/653 R
`5,050,608
`5,436,638
`7/1995 Bolas et al.
`345/156
`... 340/706
`. 345/161
`7/1995 Reeves..
`5,065,145
`11/1991 Purcell
`........
`5,436,640
`
`
`.
`364/167.01
`8/1995 Taylor ...
`5,072,361
`12/1991 Davis et al.
`5,445,166
`. 128/897
`
`
`1/1992 Stamperet al.
`oo. 463/31
`9/1995 Massiminoct al.
`. 340/407.1
`5,080,377 *
`5,451,924
`2/1992 McMurtry et al. o.. 364/474.03
`10/1995 Jacobusetal. .....
`318/568.11
`5,088,046
`5,459,382
`
`
`2/1992 Oyama oo. nceeecceeeeeeeersees 364/560
`11/1995 McMahonetal.
`5,088,055
`5,467,763
`600/201
`.... 340/710
`5,471,571 * 11/1995 Smith et al.
`.....
`5,095,303
`3/1992 Clark etal.
`345/437
`
`
`4/1992 MeIntosh oo. eee 318/568.22
`1/1996 Reddyet al.
`5,103,404
`5,482,051 *
`600/546
`
`
`4/1992 ROSCI voces ceeeeteccrescrecee 200/6
`wee 345/156
`5,107,080
`5,512,919
`4/1996 Araki
`.
`
`.. 341/22
`364/167.01
`5,107,262
`4/1992 Cadoz et al.
`.
`5,513,100
`4/1996 Parker
`
`5/1992 Moncrief et al... 273/448 B
`5,116,051
`5,526,480
`6/1996 Gibson......
`wee 395/154
`5,126,948
`6/1992 Mitchell et al. ww. 364/474.03
`5,530,455
`6/1996 Gillick et al. eee 345/163
`
`

`

`US 6,219,032 B1
`
`Page 3
`
`OTHER PUBLICATIONS
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`5,565,840 * 10/1996 Thorneret al. oe 340/407.1
`5,565,887
`10/1996 McCambridgeetal. ............ 345/145
`Su, S. Augustine et al., “The Virtual Panel Architecture: A
`10/1996 Selker wens
`wa 345/146
`5,565,888
`
`3D Gesture Framework,” IEEE 1993, pp. 387-393.
`.. 345/157
`5,570,111
`10/1996 Barrett et al.
`....
`
`Jacobsen, S.C. et al., “High Performance, High Dexterity,
`11/1996 Rosenbergetal.
`5,576,727
`345/179
`Force Reflective Teleoperator II,” ANS Topical Meeting on
`5,577,981 * 11/1996 Sarvik veescecsecssecsseesseessesesesesecesee 482/4
`Robotics & Remote Systems, Albuquerque, New Mexico
`5,583,478 * 12/1996 REMZi oo. eeceeeeseseeetenene 340/407.1
`
`Feb. 24-27, 1991, pp. 1-10.
`«. 364/578
`5,587,937
`12/1996 Massieetal.
`Kotoku, Tetsuo et al., “Environment Modeling for the Inter-
`5,589,828 * 12/1996 Armstrong 0... TAIAT1 XY
`active Display (EMID) Usedin Telerobotic Systems,” IEEE
`5,589,854
`6/1995. Tsai vescssssssesscssssusssssseeseen 345/161
`Nov. 3-5, 1991, pp. 99-1004.
`» 345/145
`5,596,347
`1/1997 Robertsonetal.
`
`Bejezy, Antal K., “The Phantom Robot: Predictive Displays
`ee .
`4,hogs Rsunine Ct AL.
`soeesreeseeen oa05,00
`*
`*
`.
`2
` ...cecesecceececerteeereeenes
`>
`9
`OSCTIDOTYS
`orpucoperation with Time Delay,”
`IEEE 1990, pp.
`5623642 *
`4/1907 Katz et ai -
`364)578
`
`.
`.
`4/1997 Massie et al. oc... eeeeeceeeeeeee 364/678
`5,625,576
`Buttolo, Pietro et al., “Pen—Based Force Display for Preci-
`5,629,594
`5/1997 Jacobusel ale
`ccsssesesseenee 318/568.11
`sion Manipulation in Virtual Environments,” IZEL Mar.
`5,631,861 *
`5/1997 Kramer ou...
`.. 364/806
`1995, pp. 1-8.
`5,642,469 *
`6/1997 Hannaford etal.
`w. 345/179
`Tan, Hong Z. et al, “Human Factors for the Design of
`5,643,087 *
`7/1997 Marcusetal. ...
`. 463/38
`
`Force—Reflecting Haptic Interfaces,” Tan, Srinivasan, Eber-
`+ 345/161
`9/1997 Culver.......
`5,666,138
`
`man, & Chang, ASME WAM 1994,pp. 1-11.
`» 345/420
`5,666,473
`9/1997 Wallace .
`Ellis, RE.etal, “Design and Evaluation of a High-Perfor-
`700,219 * 1/1998 Chenetal.2600/505
`mance Prototype Planar Haptic Interface,” ASME Dec. 3,
`5.714.978 *
`2/1908 Yamanaka etal.
`_. 345/157
`
`1993, Dsc—vol. 49, pp.55—-65.
`2/1998 Rosenberget al.
`.. 345/161
`5,721,566
`
`Adelstein Bernard D. et al., “A High Performance Two
`w. 345/156
`5,731,804
`3/1998 Rosenberg.
`...
`
`Degree—of—Freedom Kinesthetic Interface,” Massachusetts
`3/1998 Rosenbergetal.
`.. 345/161
`5,734,373
`
`Institute of ‘lechnology 1992, pp. 108-112.
`...
`.. 345/173
`5,736,978
`4/1998 Hasseret al.
`Colgate J. Edward et al., Implementation of Stiff Virtual
`4/1998 Rosenberg et al. oe 345/161
`5,739,811
`Walls in Force—-Reflecting Interfaces, Sep. 22, 1993.
`3742,278 *
`4/1998 Chen et al. oer 345/156
`Iwata, Hiroo et al., Volume Haptization,
`IEEE 1993,
`“ seer
`25003 +
`si1908 peeroeal
`
`PP-16-18.
`a
`S/1998 OSga cessessssssessesnssessssseeee 345/146
`5,757,358 *
`Fischer, Patrick et al., “Specification and Design of Input
`.. 345/161
`5,767,839
`6/1998 Rosenberg
`
`devices for Teleoperation,” 1990.
`5,771,037.
`6/1998 Jackson.....
`w 345/157
`Burdea, Grigore et al., “Distributed Virtual Force Feed-
`5,781,172
`7/1998 Engel et al.
`«. 345/164
`
`back,” IEEE, May 2, 1993, pp. 25-44.
`7/1998 Keyson ........
`.. 345/167
`5,784,052
`Rosenberg, Louis B., “The Use of Virtual Fixtures as Per-
`7/1998 Breweret al....
`» 345/339
`5,786,818
`ceptual Overlays to Enhance Operator Performance in
`8/1998 Salcudeanet al. oe 345/184
`5,790,108
`:
`At
`:
`5,805,140
`9/1998 Rosenberg et al. 345/161
`Remote Environments,” Air Force Material Command, Sep.
`_ 345/348
`5.805.165 *
`9/1998 Thorne, III etal
`
`......
`345/145-1992, pp. 142.
`5,808,601 *
`9/1998 Leahet al.
`
`.
`5,818,423 * 10/1998 Pugliese et al.
`345/157
`Rosenberg, Louis B., The Use of Virtual Fixtures to Enhance
`
`essences 341/20
`5,825,308
`10/1998 Rosenberg. 00...
`Operator Performance in Time Delayed Teleoperation, Arm-
`5,844,392
`12/1998 Peurachet al.
`. 318/568.17
`strong Laboratory, Mar. 1993, pp. 1-45.
`3/1999 Redlich ....-ssecee seen 345/163
`5,877,748
`Rosenberg, Louis B., “Perceptual Design of a Birtual Rigid
`5,889,670
`3/1999 Schuler et al. voeee 364/186
`Surface Contact,” Center for Design Research Stanford
`
`5,956,484
`9/1999 Rosenberget al.
`- 395/200.33
`University, Air Force material Command, Apr. 1993
`2/2000 Rosenberg et al.ce 345/157
`6,020,876
`,
`?
`,
`6,028,593
`2/2000 Rosenberg etal. saceeccesnen 345/156
` PP-1-41.
`'
`
`... 345/156
`6,057,828
`5/2000 Rosenbergetal.
`Rosenberg, Louis B. et al., “Perceptual Decomposition of
`
`«..sssusssccssssssecsseseee 341/20
`6,061,004
`5/2000 Rosenberg.
`Virtual haptic Surfaces, ” IEEE, Oct. 1993.
`w- 345/145
`6,078,308
`6/2000 Rosenberget al.
`Rosenberg, Louis B., “Virtual Fixtures as Tools to Enhance
`7/2000 Rosenberg oo... cece 345/156
`6,088,019
`Operator Performance in Telepresence Environments,” SPIE
`telemanipulator Technology, 1993.
`Rosenberg, Louis B., “Virtual Haptic Overlays Enhance
`Performance in Telepresence Tasks,” SPIE 1994.
`Burdea, Grigore et al., “Dextrous Telerobotics with Force
`Teedbacks—An Overview,” Robotica 1991, vol. 9.
`Colgate, J. Edward et al., “Implementation of Stiff Virtual
`Walls in Force—Reflecting Interfaces,” 1993, pp. 1-9.
`Yamakita, M. et al., Tele—Virtual Reality of Dynamic
`Mechanical Model, IEEE Jul. 7-10, 1992, pp. 1103-1110.
`Aldstein, Bernard D. et al., “Design and Implementation of
`a Force Reflecting Manipulandum for Manual Control
`Research,” 1992, pp. 1-24.
`Ouh-young, Ming et al., “Force Display Performs Better
`than Virtual display in a Simple 6—D Docking Task,” IEEE
`1989, pp. 1462-1466.
`
`10/1992
`2/1992
`of 0000
`of 0000
`of 0000
`1/1992
`11/1994
`1/1995
`8/1995
`8/1995
`5/1996
`7/1996
`9/1996
`6/1997
`
`2254911A
`4-34610
`W09502233
`W09532459
`W09642078
`WO09200559
`0626634.A2
`WO9502801
`W09520787
`WO09520788
`W09616397
`W09622591
`W09628777
`W09720305
`
`(GB).
`(IP) ,
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`(WO) .
`
`

`

`US 6,219,032 B1
`
`Page 4
`
`Kim, WonS. et al., “Graphics Displays for Operator Aid in
`Telemanipulation,” IEEE 1991, pp. 1059-1067.
`Hannaford, Blake et al., “Performance Evaluation of a
`Six-Axis Generalized Force—Reflecting Teleoperator,”
`TEEE May/Jun. 1991, vol. 21, No. 3, pp. 620-633.
`Kim, WonS. et al., A Teleoperation Training Simulator with
`Visual and Kinesthetic Force Virtual Reality.
`Burdea, Grigore et al., “A Portable Dextrous Master with
`Force Feedback,” Presence: ‘leleoperators and Virtual Envi-
`ronments, MIT Press, Jun. 1991.
`Fisher, S.S. et al., “Virtual Environment Display System,”
`ACM Interactive 3D Graphics, Oct. 1986.
`“The Personal Digitizer™,” Immersion Human Interface
`Corporation 1994.
`“High Performance Model of the Immersion Probe,” Immer-
`sion—Probe-MD™, Immersion Human Interface Corpora-
`tion.
`3D HumanInterface Tool, Immersion Probe™, Immersion
`HumanInterface Corporation 1994.
`Meyer, Kenneth etal., “A Survey of Position Trackers,” The
`Massachusetts Institute of Technology 1992, Presence,vol.
`1, No. 2.
`“Useful Technologyfor Your Idea File,” Design News, Mar.
`7 1994, pp. 63.
`It Palpable Progress,” Business
`Smith, Geoffrey, “Call
`Week, Oct. 9, 1995, p. 93, 96.
`Minsky, Margaret et al., “Feeling and Seeing: Issues in
`Force Display,” ACM 1990, pp. 235-242.
`Herndon , J.N. et al., “The State-of-the-Art Model M-s
`Maintence System,” Proceedings of the 1984 National Topi-
`cal Meeting on Robotics and Remote Handling in Hostile
`Environments, American Nuclear Society, pp. 59-65.
`Batter, JamesJ. et al., “Grope—1: A Computer Displayto the
`Sense of Feel,” pp. TA-a—188—Ta—4-192.
`Gotow, J.K., et al., “Perception of Mechanical Properties at
`the Man-Machine Interface,” IEEE 1987, pp. 688-689.
`“Proceedings of the IFIP Congress 65,” International Fed-
`eration tor Information Processing, Information Processing
`1965, vol. 3, New York, May 24-29, pp. 506.
`Atkinson, William D. et al., “Computing with Feeling,”
`Comput. & Graphics, vol. 2, No. 2—-E, pp. 97-103.
`Noll, A. Michael, “Man-Machine Tactile Communication
`Dissertation,” Polytechnic Institute of Brooklyn, Jun. 1971,
`pp. 1-88.
`Ouh-young, Ming, “Force Display in Molecular Docking”
`Chapel Hill 1990, pp. 1-85.
`Ouh-young, Minget al., “Using A Manipulator for Force
`Display in Molecular Docking,” IEEE 1988, pp. 1824-1829.
`“Foot—Operated Mouse,” IBM Technical Disclosure Bulle-
`tin, Apr. 1986, vol. 28, No. 11.
`Iwata, Hiroo, “Pen—based Haptic Virtual Environment,”
`Institute of Engineering Mechanics, University of Tsukuba,
`Japan, pp. 287-292.
`Russo, Massimo Andrea, “The Design and Implementation
`of a Three Degree-of-Freedom Force Output Joystick,”
`Department of Mechanical Engineering, May 11, 1990, pp.
`9-40 & 96 & 97.
`
`Tan, Hong Z et al., “Manual Resolution of Compliance
`When Work and Force Cues are Minimized,” ASME 1993,
`DSC-vol. 49, pp. 99-104.
`Howe, Robert D., “Task Performance with a Dextrous
`Teleoperated Hand System,” Proceedings of SPIE, Nov.
`1992, vol. 1833, pp. 1-9.
`
`Snow,E. et al., “Compact Force—Reflecting Hand Control-
`ler,” JPL, Apr. 1991, vol. 15, No. 3, Item No. 153, pp. 1-15a.
`Schmult, Brian et al., “Application Areas for a force—Feed-
`back Joystick,” ASME 1993, DSC—vol. 49, pp. 47-54.
`Hasser, Christopher John, “Tactile Feedback for a Force—Re-
`flecting Haptic Display,” The School of Engineering, Uni-
`versity of Dayton, Dec. 1995, pp. ii—xii & 1-96.
`Precision Machine Design, Slocum, Prentice Hall, pp. 661,
`664.
`
`Kelle et al. “On the Development of a Force—Feedback
`Mouse andIts Integration into a Graphical User Interface,”
`1994 Int’l Mechanical Engineering Congress and Exhibi-
`tion, Chicago, Nov. 1994.
`Kelley, A.J. et al., “MagicMouse: Tactile and Kinesthetic
`Feedback in the Human—ComputerInterface using an Elec-
`tromagnetically Actuated Input/Output Device,” Univ. of
`British Columbia, Dept. of Electrical Engineering ‘lech,
`Report, 1993, pp. 1-27.
`Hannaford, B. et al., “Force Feedback Cursor Control ,”
`NASATech Brief, vol. 13, No. 11, Item #21, 1989, pp. 1,
`144.
`
`Ouh-young,M. et al., “Creating an Ilustion of Feel: Control
`Issues in Force Display,” Computer Science Dept., Univ. of
`N. Carolina, 1989, pp. 1-14.
`Ramstein, C. et al., “The Pantograph: A Large Workspace
`Haptic Device for a Multimodal Human-ComputerInterac-
`tion,” Computer-Human Interaction, CHI ’94. 1994.
`Millman, P, “Design of
`a Four Degree—of-freedom
`Force—Reflecting Manipulandum with a Specified Force/
`Torque Workspace,” Proc. of the 1991 IEEE Int’l Conf. on
`Robotics and Automation, 1991, pp. 1488-1493.
`Hirota, K., “Development of Surface Display”,
`0-7803-1363, 1993, pp. 256-262.
`Ansley, D., “Thimble gets in touch with reality,” New
`Scientist, 1994, p.19.
`Brooks, F. et al., “Project GROPE—Haptic Displays for
`Scientific Visualization,” Computer Graphics, vol. 24, Num-
`ber 4, Aug. 1990, pp. 177-185.
`Iwata, H. “Artificial Realtiy with Force—feedback: Devel-
`opment of Desktop Virtual Space with Compact Master
`Manipulator”, Computer Graphics, vol. 24, No. 4, Aug.
`1990, pp. 165-170.
`Kilpatrick, P., “The Use of a Kinesthetic Supplement in an
`Interactive Graphics System”, Univ. of North Carolina,
`1976.
`
`IEEE
`
`Yokokohyji et al., “What You Can See is What You Can Feel
`—Development of a Visual/Haptic Interface to Virtual Envi-
`ronment”, Proceedings of VRAIS °96,
`IEEE 1996, pp.
`46-54.
`
`Krishna Roy, “Virtual Presence Takes Surgeons through the
`Virtual Keyhole to Hone Their Skills’, Business & Industry,
`Jul. 4, 1995.
`Akamatsu et al., “Multimodal Mouse: A Mouse—Type
`Device with Tactile and Force Display”, Presence, vol. 3,
`No. 1, Winter 1994, pp. 73-80.
`Rosenberg, L., “Virtual Fixtures’: Perceptual Overlays
`Enhance Operator Performance in Telepresence Tasks”,
`Stanford University, Jun. 1994, pp. 1-214.
`Winey III, C., “Computer Simulated Visual and Tactile
`Feedback As An Aid to Manipulator and Vehicle Control”,
`MIT, 1981, pp. 1-80.
`
`* cited by examiner
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 1 of 27
`
`US 6,219,032 B1
`
`HOST COMPUTER SYSTEM
`yode
`I13ee 1]
`
`
`SYSTEM
`AUDIO OUT-|
`!
`|
`CLOCK
`PUT DEVICE
`!
`!
`
`
`!
`!
`\
`'
`'
`1
`I
`|
`!
`|
`
`t
`
`
`
`y10
`
`HEAR
`
`]
`|
`|
`_VIEW _|
`
`Ww
`
`ee
`
`lt
`
`FORCE FEEDBACK
`
`
`INTERFACE DEVICE
`
`
`ACTUATOR
`
`
`INTERFACE
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 2 of 27
`
`US 6,219,032 B1
`
`MICRO-
`PROCESSOR
`
`26
`
`Figure 2
`
`wo 88
`
`“44
`
`+5V
`
`48
`
`R2
`
`R1
`
`DAC
`
`OUTPUT
`
`-V
`
`52
`
`-46
`
`lout
`
`L_{
`
`ACTUATOR
`
`57
`
`R4
`
`R3
`
`

`

`U.S. Patent
`
`Apr. 7, 2001
`
`Sheet 3 of 27
`
`US 6,219,032 B1
`
`MICRO-
`PROCESSOR
`
`eo i = ele EE EE ee ee ee ee eee ee ee ee ee ee ee eee ee
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 4 of 27
`
`US 6,219,032 B1
`
`v0
`
`™~
`
`<a
`
`
`
`POWER-UP HOST
`COMPUTER AND
`INTERFACE DEVICE
`
`MICROPROCESSOR
`
` HOST COMPUTER
`INTERFACE DEVICE
`
`y
`76
`PROCESS HOST
`APPLICATION
`
`86
`
`y
`ACTIVATE
`
`
`
`
`RECEIVE
`88|READ RAW
`
`
`SENSOR DATA
`\] DATA FROM
`
`FROM MICRO-
`
`SENSORS
`
`PROCESSOR
`
`
`PROCESS RAW
`DATA INTO
`
`SENSOR DATA
`
`
`
`
`CHANGE
`
`SEND SENSOR
`INFORCE
`
`
`DATA TO HOST
`ON OBJECT RE-
`QUIRED?
`
`
`
`
`DETERMINE LOW-LEVEL
`
`FORCE COMMAND IN
`ACCORDANCE WITH A
`SELECTED REFLEX
`
`
`
`PROCESS, SENSOR DATA,
`
`HOST APPLICATION, AND
`
`OTHER DATA
`OUTPUT
`
`LOW-LEVEL
`
`FORCE
`COMMAND TQ
`ACTUATORS
`
` SEND LOW-LEVEL
`
`
`FORCE COMMAND TO
`
`MICROPROCESSOR
`
`Figure 4
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 5 of 27
`
`US 6,219,032 B1
`
`100
`
`\
`
`HOST COMPUTER
`
`POWER-UP HOST
`COMPUTER AND
`INTERFACE DEVICE
`
`MICROPROCESSOR
`
`ACTIVATE
`INTERFACE DEVICE
`
` i
`
`PROCESS HOST
`
`(08
`(starr)
`
`
`
`APPLICATION
`
`
`
`
`
`
`778|READ RAW
`READ SENSOR
`\) DATA FROM
`DATA OF USER
`
`
`
`OBJECT FROM
`
`SENSORS
`
`MICRO-
`
`PROCESSOR
`
`PROCESS RAW
`
`DATA INTO
`
`SENSOR DATA
`
`
`
`STORE AND
` CHANGE
`
`SEND SENSOR
`N TYPE OF FORC
`
`
`
`DATA TO HOST
`ON OBJECT RE-
`
`
`QUIRED?
`127
`
`RECEIVE
`NEW HIGH
`
`
`LEVEL FORCE
`
`
`DETERMINE
`OMMAND?
`
`
`HIGH LEVEL
`
`
`COMMAND
`SELECT REFLEX
`PROCESS BASED
`ON HIGH LEVEL
`
`
`FORCE
`
`ISSUE HIGH LEVEL
`
`
`COMMAND
`FORCE COMMAND TO
`
`MICROPROCESSOR
`DETERMINE FORCE
`
`COMMAND TO
`ACTUATORS IN
`
`
`
`
`ACCORDANCE WITH
`REFLEX PROCESS,
`SENSOR/OTHER DATA,
`AND HIGH LEVEL
`
`FORCE COMMAND
`
`
`OUTPUT LOW-LEVEL
`
`FORCE COMMAND
`
`Figure 5
`
`
`
`
`
`
`
`7128
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 6 of 27
`
`US 6,219,032 B1
`
`
`
`
`Figure 6
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 7 of 27
`
`US 6,219,032 B1
`
`
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 8 of 27
`
`US 6,219,032 B1
`
` yy. 186b,
`
`|F
`
`189d
`ya os cS,
`a
`
`184a
`
`
`
`
`188a Se
`=
`
`~
`
`tL
`
`yj
`
`188b
`ar M
`
`\ 189c 186a
`LEE 2 oo
`EP
`
`
`
`l
`_- fe
`2 foe
`
`
`|
`190°
`|
`Tf
`
`
`
`190
`
`Figure &
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 9 of 27
`
`US 6,219,032 BI
`
`ore7
`
`
`
`106-7zoc7
`
`SSVTO8ENs
`
`
`
`SYSLSNVeVdGNVGNVWWODJO3JIdWVxXa390uOS
`
`
`
`
`
`SGNVWAOSIONLNODALVY
`
`NOILIGNOS
`
`NOILIGNOD
`
`NOILIGNOD
`
`NOILIGNOD
`
`AV1YsA0
`
`AV1ddAO
`
`AV1dSAO
`
`[AWTasAAO
`
`NOILIGNOS
`
`AV18dSAQ
`
`
`
`
`
`
`‘NOILOSHNIC'NOILVYENG‘JGNLINDVW)SALVIAANOILVUSIA
`
`
`B0EN,NolandJGNLINDVWW)JONOsONIYOLSAa
`
`
`
`
`‘NOILOSUIC‘NOILVYNG'SGNLINDVW)3DYNO42NOLIN
`
`
`
`
`(FIALS‘NOILVYNG‘JGNLINDVW)J1SVLSNNAIgVLSNN
`
`
`
`(NOLLOANIC‘NOLLVYNA‘AGNLINSVA)L10rLor
`
`
`
`
`(FTALS‘NOILVYNG‘AGNLINSVYW)37agomMAIddOM
`
`
`
`(AIALS‘NOILVANG‘LN3AIOIS4S5OO)9N1SHSONS
`
`
`‘NOILVYNG‘LNAIOI4SAOO)ONINdS4ONIYOLSAY
`
`
`‘NOILVYUNG‘JGNLINDVW)JONO4AYOLOAA
`
`
`(GNVEGV4d‘JIALSONIYdS
`(ANVddvad“FTALS7~GOE
`
`
`
`(NOILOSNIG
`
`
`
`(ATALS‘AONSNDAYSA
`
`
`
`(AYISOLNV‘'NOLLNG
`
`6ANS
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 10 of 27
`
`US 6,219,032 B1
`
`312
`
`DISPLACEMENT
`
`FORCE
`-
`e816
`
`
`f,
`figure 1a
`
`
`DISPLACEMENT
`
`Figure 10D
`
`
`
`
`DISPLACEMENT
`
`Figure 10c
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 11 of 27
`
`US 6,219,032 B1
`
`FORCE
`
`320.
`
`O
`
`DISPLACEMENT
`
`Figure Ila
`
`FORCE
`
`322
`
`O
`
`DISPLACEMENT
`
`Figure 11D
`
`
`324
`
`FORCE
`
`
`
`
`DISPLACEMENT
`
`Figure IIc
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 12 of 27
`
`US 6,219,032 B1
`
`326ae
`
`X
`
`Figure 12
`
`
`
`TIME
`
`Figure 15a
`
`FORCE
`
`
`
`TIME
`
`Figure 13D
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 13 of 27
`
`US 6,219,032 BI
`
`ges7
`
`(GYISOLNY‘NOLLNG‘NOILOFYIC‘JGNLINDVW)NOLLNA
`
`
`
`
`SAOOND SYAaLAWNVaVdGNVGNVAWIWODAOAldWVXaA
`(AONVLSIG-d¥NS‘NOILVOOT‘SGNLINDVW)YadavX
`
`
`
`(ASNHS‘SJONVIIdWIOD'ALIAVYD‘SSVW)J1d0vd
`
`
`
`
`(JONVISICG-dVNS‘JTALS‘SGNLINSVW)
`
`
`(JONVLSIC-dVNS‘FTALS‘AGNLINDVW)LOAIG
`
`
`
`(JTALS‘ALISNAG'SCQNLINDVA)SYNLXSL
`
`
`
`(ASNAS‘NIDIMO‘JGNLINDVW)G7a!4
`
`
`
`(NOILOJYIC‘JGNLINDVW)JONOSd\|
`
`
`SGNVWWODTIOYLNODNOILISOd
`
`
`
`plNol]
`
`YOLOAA
`
`AAQOYD
`
`LOAIC
`
`FaNLXsal
`
`dsldavd
`
`Q1aAld
`
`41ddqvd
`
`NOLLNG
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 14 of 27
`
`US 6,219,032 B1
`
`FORCE
`349
`o™
`
`
`
`
`Ss
`
`DISPLACEMENT
`
`Figure 15
`
`DISPLACEMENT
`
`THICKNESS
`
`
`
`Figure 16
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 15 of 27
`
`US 6,219,032 B1
`
`0,
`
`Fig. 17
`
`290 DY!
`Fig. 17b
`aN)amFig. 17¢
`Fig. I7d
`
`350
`
`352
`
`Fig. 17e
`330- x t
`OD) Fig. 17f
`wy(Qt Fig. I7g
`woas Fig. 17h
`
`352
`
`By
`
`Fig. 171
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 16 of 27
`
`US 6,219,032 B1
`
`Figure 17k
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 17 of 27
`
`US 6,219,032 B1
`
`
`
`Figure 18
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 18 of 27
`
`US 6,219,032 BI
`
`
`
`
`
`
`Properties
`
`Run
`Exit Windows
`
`
`
`
`
`File|Options Window Help
`
`
`
`
`
`
`
`Options Window
`
`__Helpf
`¢
`
`Figure 19
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 19 of 27
`
`US 6,219,032 B1
`
`
`
`970 == EXTERNALFORCE ~~~~~_
`ooETure CAPTURE
`08 a
`FORCE
`oH 577
`
`a
`
`579
`
`cf
`
`_ \
`
`mS
`5557 ~~
`
`574
`
`208
`
`Figure 20b
`
`506°”
`a
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 20 of 27
`
`US 6,219,032 BI
`
`
`
`Figure 20c
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 21 of 27
`
`US 6,219,032 B1
`
`507
`
`517
`505
`
`[7|ProgramManager|=|
`| File
`Options} Window
`Help
`l
`
`502
`
`
`
`
`SREERR!
`
`Properties
`Run
`Exit Windows
`
`510
`
`N
`
`Da&Go
`
`Nn©oS
`
`1S)x
`
`t
`
`Startup
`+
`
`RESETSSA
`
`Figure 21
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 22 of 27
`
`US 6,219,032 B1
`
`
`
`File Options Window
`
`Help
`
`2)
`
`RSXREQ)
`
`
`
`Startup
`
`
`
`Figure 22
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 23 of 27
`
`US 6,219,032 B1
`
`610 ~
`
`
`
`
`
`
`CALIBRATE
`USER OBJECT
`POSITION
`
`614
`616
`UPDATE DISPLAY
`
`
`
`MAP FORCES TO
`GRAPHICAL
`OBJECTS IN GUI
`
`
`
` READ USER OBJECT
`
`POSITION AND
`
`OF LOWEST
`HIERARCHY IN
`WHICH CURSOR
`IS LOCATED
`
`
` DETERMINE TARGET
`
`
`
`
`
` APPLY APPROPRIATE
`
`FORCE TO USER
`
`OBJECT BASED ON
`
`SELECTED TARGET
`
`Figure 23
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 24 of 27
`
`US 6,219,032 B1
`
`we
`
`676
`
`oF
`
`634
`
`YES
`
`ASSIGN RADIAL
`DEAD, CAPTURE,
`AND EXTERNAL
`FORCE RANGES
`
`1-944
`
`646
`
`DEAD AND CAPTURE
`RANGES, AND RADIAL
`EXTERNAL FORCE
`RANGES
`
`ASSIGN RADIAL
`INTERNAL AND
`
`648
`
`
`
`
`
`
`
`ASSIGN RECTANGULAR
`OR LINEAR DEAD,
`RECTANGULAR CAPTURE,
`AND RECTANGULAR
`
`
`
`
`
`YES
`
`YES
`
`YES
`
`VES
`
` ASSIGN RECTANGULAR
`
`
`EXTERNAL RANGES
`
`EXTERNAL RANGES 656
`
`ASSIGN SPECIAL
`
` ASSIGN
`
`MAGNITUDE TO
`MAGNITUDE
`
`
`FORCE RANGES
`ACCORDING TO
`
`
`
`
`
`TARGET TYPE
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 25 of 27
`
`US 6,219,032 B1
`
`oye
`
`620x
`
`
`
`
`
`
`FOLLOW
`HIERARCHY TO
`LOWEST TARGET
`CURSOR IS IN
`
`662
`
`
`
`
`Figure 25
`
`664
`
`
`
`CURSOR
`IS IN MORE
`
` SELECT TARGET
`THAN ONE TARGET
`OF LOWEST
`LEVEL?
`
`668
`
`SELECT TOP
`TARGET
`
`666
`
`

`

`U.S. Patent
`
`Apr.17, 2001
`
`Sheet 26 of 27
`
`US 6,219,032 B1
`
`622
`
`fo
`
`672
`
`CALCULATE INTERNAL FORCE
`BASED ON FUNCTION HAVING
`APPROPRIATE PARAMETERS
`
`4
`
`Figure 26
`
`
`
`
`
`
`
`
`TOTAL FORCE =
`INTERNAL FORCE OF
`
`LOWEST TARGET
`
`
`
`
`DETERMINE CHILDREN
`THAT WILL AFFECT
`USER OBJECT
`
`675
`
`
`
`680
`
`
` COMMAND
`
`EXTERNAL
`
`
`YES
`GESTURE IS BEING
`FORCE OF AN
`
`
`INPUT THAT WOULD AFFECT
`
`
`UNVISITED CHILD
`
`FORCE ON USER
`
`
`NOT ADDED?
`OBJECT?
`
`
`
`ADJUST TOTAL
`
`
`
`SELECT A CHILD
`FORCE BASED
`
`ON COMMAND
`AND COMPUTE
`
`
`GESTURE
`EXTERNAL
`
`
`
`FORCE
`
`OTHER
`
`
`CONDITION THAT
`YES
`
`TOTAL FORCE =
`
`
`AFFECTS
`
`
`
`FORCE ON USER
`TOTAL FORCE +
`OBJECT?
`EXTERNAL FORCE
`
`
`
`OF SELECTED CHILD
`
`
`
`
`APPLY TOTAL
`
`
`FORCE TO USER
`
`
`OBJECT
`
`
`Cone)889
`686
`
`
`ADJUST TOTAL
`
`FORCE BASED
`
`ON OTHER
`CONDITION
`
`

`

`U.S. Patent
`
`Apr. 17, 2001
`
`Sheet 27 of 27
`
`US 6,219,032 B1
`
`—_ eo Figure 27
`
`N
`
`c
`
`PRESENT
`POSITION IN
`CAPTURE
`
`REGION?
`
`YES
`
`NO
`
`696
`
`
`
`WAS
`LAST NON-CAPTURE
`
`
`POSITION IN DEAD
`
`REGION?
`
`
`YES
`
`700
`
`IN DEAD
`
`
`
`NO
`
`REGION?
`YES
`FORCE
`703
`701
`
` IN
`EXTERNAL
`
`REGION?
`
`VES
`
`APPLY EXIT
`CAPTURE
`
`698
`
`NO
`
`APPLY ENTRY
`CAPTURE
`
`FORCE
`
`706
`
`APPLY
`EXTERNAL
`
`FORCE
`
`704
`
`

`

`US 6,219,032 B1
`
`1
`METHOD FOR PROVIDING FORCE
`FEEDBACK TO A USER OF AN INTERFACE
`DEVICE BASED ON INTERACTIONS OF A
`CONTROLLED CURSOR WITH GRAPHICAL
`ELEMENTSIN A GRAPHICAL USER
`INTERFACE
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation-in-part of parent US.
`patent application Ser. No. 08/566,282,filed Dec. 1, 1995 is
`now U.S. Pat. No. 5,734,373, on behalf of Louis B. Rosen-
`berg et al., entitled, “Method and Apparatus for Controlling
`Force Feedback Interface Systems Utilizing a Host
`Computer,” assigned to the assignee of this present
`application, and which is incorporated by reference herein.
`BACKGROUND OF THE INVENTION
`
`15
`
`invention relates generally to interface
`The present
`devices for allowing humans to interface with computer
`systems, and more particularly to computer systems and
`computer interface devices that provide force feedback to
`the user.
`
`2
`graphical objects or predefined regions on a computer
`screen. Such manual tasks can be described as “targeting”
`activities where a user physically manipulates a mouse,
`joystick, or other interface device in order to command the
`cursor to a desired location or displayed object, known as a
`“target” herein. Such targets can include, for example, icons
`for executing application programs and manipulating files;
`windowsfor displaying icons and other information; pull-
`down menusfor selecting particular functions of the oper-
`ating system or an application program;buttonsfor selecting
`presented options; and scroll bars or “sliders” for scrolling
`information in windows.
`
`Upon moving the cursor to the desired target, the user
`must maintain the cursor at the acquired target while press-
`ing a button, squeezing a trigger, depressing a pedal, or
`making some other gesture to command the execution of the
`given selection or operation. Examples of targeting tasks
`include positioning a cursor on a graphical icon, selecting
`and pressing a graphical representationof a button, choosing
`among numerousitems within a graphical representation of
`a pull-down menu,setting a continuous analog value from a
`provided range of values by positioning an indicator within
`a graphical representation of a scroll bar, selecting a region
`of text by highlighting a region using the cursor, as well as
`a number of other common windows-basedand text-based
`
`metaphors.
`The movementof a cursor onto various displayed graphi-
`cal objects of a GUI may require significant dexterity. Users
`may move the cursor too far over an object and have to
`backtrack their cursor. Or, particular graphical objects might
`be mistakenly selected when the user does not wish to select
`the object due to pressing a button or moving the cursor by
`accident. In addition, a user may become confused as to
`which window a cursoris positioned in if the user is viewing
`other data on the screen at the same time as moving the
`cursor.
`
`In particular, persons with neuromotor disabilities who
`suffer from spastic manual control have much greater dif-
`ficulty interacting