US006424333B1
`(10) Patent No.
`a2) United States Patent
`US 6,424,333 B1
` Tremblayetal. (45) Date of Patent: *Jul. 23, 2002
`
`
`
`(75)
`
`(54) TACTILE FEEDBACK MAN-MACHINE
`INTERFACE DEVICE
`Inventors: Mark R. Tremblay, Mountain View;
`MarkH.Yim,Palo Alto, both of CA
`US(
`)
`Immersion Corporation, San Jose, CA
`(US)
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 0 days.
`
`(73) Assignee:
`
`(*) Notice:
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21) Appl. No.: 09/838,052
`(22)
`Filed:
`Apr. 18, 2001
`Related U.S. Application Data
`
`(63) Continuation of application No. 09/561,782,filed on May 1,
`2000, now Pat. No. 6,275,213, which is a continuation of
`application No. 09/066,608,filed on Apr. 24, 1998, now Pat.
`No. 6,088,017, which is a continuation of application No.
`08/565,102, filed on Nov. 30, 1995, now abandoned.
`(SL) Unt, C1. ec eeeeeccecceceeeeeeteseeeeereeeeeneees G09G 5/00
`(52) US. Che oo eee eee 345/156; 345/702; 414/5
`(58) Field of Search oo... 345/156, 158,
`345/157, 700, 701, 702, 703; 414/1-7;
`901/32-34
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`3,919,691 A
`11/1975 Noll woe eee eee eeeee 340/172.5
`4,414,984 A
`11/1983 Zarudiansky .
`we. 128/774
`4,477,043 A
`10/1984 Repperger
`....
`ve, 244/223
`4,500,983 A
`12/1985 Williams...
`++ 340/825
`4,604,016 A
`8/1986 Joyce Lecceeeneeeccceeeneneeseeees 414/7
`4,706,294 A
`11/1987 Ouchida oe 381/109
`4,731,603 A *
`3/1988 McRaeetal.
`4,791,416 A
`12/1988 Adler wee eee eeeeees 340/712
`
`
`
`4,795,206 A
`A/L980 Ta eeecceecereeceeeeeereeeee 414/5
`4,800,721 A
`1/1989 Cemenskaetal. ............ 60/393
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`4400790 Al
`5/1995
`0085518
`8/1983
`(List continued on next page.)
`OTHER PUBLICATIONS
`
`DE
`EP
`
`Schmult, Brian et al., “Application Areas for a Force—Feed-
`back Joystick,” ASME 1993, DSC-— vol. 49, pp. 47-54.
`Howe, Robert D., “Task Performance with a Dextrous
`Teleoperated Hand System,” Proceedings of SPIE, 1992,
`vol. 1833, pp. 1-9.
`(List continued on next page.)
`Primary Examiner—Regina Liang
`(74) Attorney, Agent, or Firm—James R. Riegel; Paul M.
`Thyfault
`(57)
`
`ABSTRACT
`
`Aman-machine interface which provides tactile feedback to
`various sensing, body parts is disclosed. The device employs
`one or more vibrotactile units, where each unit comprises a
`mass and a mass-moving actuator. As the massis accelerated
`by the mass-moving actuator,
`the entire vibrotactile unit
`vibrates. ‘hus, the vibrotactile unit transmits a vibratory
`stimulus to the sensing body part to which it is affixed. The
`vibrotactile unit may be used in conjunction with a spatial
`placement sensing device which measuresthe spatial place-
`ment of a measured body part. A computing device uses the
`spatial placementof the measured bodypart to determine the
`desired vibratory stimulus to be provided by the vibrotactile
`unit. In this manner, the computing device may control the
`level of vibratory feedback perceived by the corresponding
`sensing bodypart in response to the motion of the measured
`body part. The sensing body part and the measured bodypart
`may be separate or the same bodypart.
`
`18 Claims, 20 Drawing Sheets
`
`a
`
`4
`
`MONITOR
`
`S
`
`“
`2109
`2708
`S
`S
`NS
`SS
`709
`
`.
`
`
`
`
`
`
`
`
`VIRTUAL SIMULATIONS
`rie ee
`TTA
`VIRTUAL
`1
`MEASURED
`-—“gopy PART
`||
`(2707)
`
`SicNAL
`(2708
`i
`GNAL
`'
`(2721)
`stenaL
`VIRTUAL
`1,
`2713
`=
`<=
`Al
`
`/
`DRIVER
`PROCESSOR
`SITE
`i!
`SENSING
`bel i 9 7 nvarne
`BODY PART
`i
`
`
`2708
`\
`i |
`SIGNAL
`iL
`compute yy
`FacTiLe
`LAS)
`COMPUTERTSL
`1
`|
`CL Lot} 2704
`VISRATION
`Tt
`2712-2715 2714
`po —— THYSICAL_SIMULATION| 5!
`a7o1 7 ‘2702
`27 ong
`2718
`|
`2700
`2711
`| 2720|||! (
`
`
`
`
`
`
`(2716 \ | fstare
`C
`||
`
`SEN
`
`||& paysae PHYSICAL |
`PHYSICAL
`|
`STATE
`i
`MEASURED STATE
`SIGNAL
`|
`|
`|
`BODY PART
`1
`Lowea. J
`t
`\.2723
`
`H
`
`
`
`2710
`
` Valve Exhibit 1052
`
`Valve Exhibit 1052
`Valve v. Immersion
`Valve v. Immersion
`
`

`

`US 6,424,333 B1
`
`Page 2
`
`6,004,134 A
`6,088,017 A *
`6,104,158 A
`6,184,868 B1
`6,198,206 B1
`6,211,861 Bl
`6,275,213 Bl *
`RE37,374 E
`
`12/1999 Marcus et al. oo. 434/45
`7/2000 Tremblay et al.
`........... 345/156
`8/2000 Jacobuset al.
`318/568.11
`
`........... 345/161
`2/2001 Shahoian et al.
`3/2001 Saarmaaet al.
`....c.c... 310/340
`4/2001 Rosenberg etal.
`. 345/163
`8/2001 Tremblay etal.
`. 345/156
`9/2001 Roston etal.
`...
`. 318/561
`
`
`
`..
`
`FOREIGN PATENT DOCUMENTS
`
`
`
`4/1989 Culver... 7A/A71.
`4,823,634 A
`
`«340/710
`9/1989 Affinito et al.
`.
`4,868,549 A
`12/1989 Embach.............eceeeeee 340/407
`4,885,505 A
`8/1990 Moncrief et al.
`........... 364/578
`4,949,119 A
`
`. 318/685
`1/1991 Lehmer
`.............
`4,983,901 A
`
`9/1991 Behenskyetal.
`.
`434/45
`5,044,956 A
`.. 244/228
`12/1991 Ferranti et al.
`....
`5,076,517 A
`4/1992 McIntosh
`5,103,404 A *
`4/1992 Cadoz etal. on 341/22
`5,107,262 A
`4/1988
`0265011
`EP
`9/1992 Hollis, Jr. et al. wc... 395/275
`5,146,566 A
`1/1990
`0349086 Al
`EP
`2/1993 Kramer
`5,184,319 A *
`
`
`
`5,185,561 A 0626634 A2=5/19942/1993 Good et al. .occcee 318/432 EP
`2/1993 Rohen .......
`.. 434/114
`5,186,629 A
`EP
`0607580 Al
`7/1994
`
`5,193,963 A
`3/1993 McAllee elal.
` 414/5
`GB
`2254911 A
`10/1992
`
`.......
`5,203,563 A
`4/1993 Loper, III
`273/148
`JP
`S62-194389
`12/1987
`5,209,661 A
`5/1993 Hildreth et al.
`oo... 434/45
`JP
`4008381
`1/1992
`5,220,260 A
`6/1993 Schuler oo... 318/561
`WO
`WO92/00559
`1/1992
`6/1993 Radke et al. vvceeecue. 318/568
`5,223,776 A
`WO
`WO96/09695
`3/1996
`5,286,203 A
`2/1994 Fulleret al.
`....
`434/45
`wo
`WO 01/03105
`1/2001
`5,296,871 A
`3/1994 Paley oo. e eee cece ee 345/163
`5,354,162 A
`10/1994 Burdea tal 414/5
`OTHER PUBLICATIONS
`2300 A
`too Copperman tl at
`. tee
`Russo, Massimo Andrea, “The Design and Implementation
`
`
`.....
`318/566
`5,381,080 A
`1/1995 Schnell et al.
`of a Three Degree-of—Freedom Force Output Joystick,”
`... 434/114
`5,388,992 A
`2/1995 Franklin etal.
`Department of Mechanical Engineering, May 11, 1990, pp.
`
`.. 345/161
`5,396,266 A
`3/1995 Brimhall.........
`9-40 & 96 & 97.
`5,399,091 A
`3/1995 Mitsumoto .......c ees 434/61
`Su, S. Augustine et al., “The Virtual Panel Architecture: A
`5,405,152 A
`4/1995 Katanics et al.
`............ 273/438
`3D Gesture Framework,” IEEE 0-7803-1363-1, 1993.
`S/1995° Schuler ..........-.
`318/561
`5,414,337 A
`Hasser, Christopher John, “Tactile Feedbackfor a Force—Re-
`
`340/407.1
`9/1995 Massiminoetal.
`5,451,924 A
`flecting
`Haptic Displav.” The School of Engineering. Uni-
`
`.. 128/733
`5,482,051 A
`‘1/1996 Reddyetal. ...
`ecting
`Hapiie
`Misplay,
`NON ae
`8
`va 345/156
`4/1996 Araki esses
`5,512,919 A
`Versity of Dayton, Dec. 1995,pp. iii-xii &1-96.
`5,513,100 A
`4/1996 Parkeretal.
`. 364/167.01
`Ellis, R.E. et al., “Design and Evaluation of a High—Perfor-
`
`8/1996 Meredith ....... eee 463/37
`5,542,672 A
`mance Prototype Planar Haptic Interface,” ASME Dec. 3,
`5,559,432 A
`9/1996 LOgUe ceeeeesscsteestesteses 324/207
`1993, DSC-vol. 49, pp. 55-64.
`
`5,965,840 A
`10/1996 Thorneret al.
`....
`340/407.1
`Burdea, Grigore et al., “A Portable Dextrous Master with
`......... 345/179
`5,976,727 A
`11/1996 Rosenberget al.
`Force Feedback,” Presence: Teleoperators and Virtual Envi-
`5,583,478 A * 12/1996 Renzi
`5,587,937 A
`12/1996 Massie et al. oo... 364/578
`Tonments, MIT Press, Jun. 1991.
`;
`12/1996 Armstrong...
`wae 341/20
`5,589,828 A
`Adlestein, Bernard D. et al., “Design and Implementation of
`
`5,589,854 A
`12/1996 Tsai
`......
`w. 345/161
`a Force Reflecting Manipulandum for Manual Control
`.
`.. 318/568
`5,629,594 A
`5/1997 Jacobuset al.
`Research,” 1992, pp. 1-24.
`
`6/1997 Hildreth etal. ....
`. 434/37
`5,634,794 A
`Minsky, Margaret et al., “Feeling and Seeing: Issues in
`
`.......... 395/99
`5,642,469 A
`6/1997 Hannatord et al.
`Torce Display,” ACM 089791-351-5, 1990, pp. 235-242.
`5,643,087 A
`7/1997 Marcuset al. 0... 463/38
`“
`:
`:
`.
`5,666,138 A
`345/161
`9/1997 Culver...
`Ouh-young, M. et al., “Creating an Tilustion of Feel: Control
`
`9/1997 Wallace .......
`. 345/420
`5,666,473 A
`Issues in Force Display,” Computer Science Dept. 1 Univ of
`5,669,818 A
`9/1997 Thorneret al.
`. 463330
`©=—-N. Carolina, 1989, pp. 1-14.
`....
`5,684,722 A
`11/1997 Thorneret al.
`...
`... 364/578
`Millman,P. et al., “Design of a Four Degree-of—Freedom
`
`.. 364/190
`5,691,898 A
`11/1997 Rosenberg etal.
`Force—Reflecting Manipulandum with a Specified Force/
`1/1998 Chen et al... 128/782
`5,709,219 A
`Torque Workspace,”
`IEEE CH2969-4,
`1991,
`pp.
`5,714,978 A
`2/1998 Yamanakaet al.
`.......... 345/157
`1488-1492.
`
`>eon ‘
`ti008 Rosenberg¢' ahve sea
`Kilpatrick, P., “The Use of a Kinesthetic Supplement in an
`5.739.811 A
`4/1998 Rosenberget al.
`........ 345/161
`Interactive Graphics System,” Univ. of N. Carolina, 1976,
`5,742,278 A
`4/1998 Chenetal. .....
`... 345/156
`pp. 1-175.
`
`
`.. 318/561
`5,754,023 A
`3/1998 Roston et al.
`..
`Akamatsu, M. et al., “Multimodal Mouse: A Mouse—Type
`5,755,577 A
`5/1998 Gillio ....
`+» 434/262
`Device with Tactile and Force Display,” Presence, vol. 3,
`6/1998 Rosenberg ....
`-- 345/161
`5,767,839 A
`No, 1, 1994, pp. 73-80.
`
`rio08 Kexcom ala- eter
`e781052 A
`Hirota, K. et al., “Development of Surface Display,” IEEE
`8/1998. Salcudean et al.
`.......-.. 345/184
`5,790,108 A
` 0-7803-1363-1, 1993, pp. 256-262.
`5,805,140 A
`9/1998 Rosenberget al.
`.. 345/161
`Atkinson, W.et al, “Computing with Feeling,” Compul. &
`5,816,823 A
`10/1998 Naimarketal. ............. 434/307
`Graphics, vol. 2, 1977, pp. 97-103.
`5,889,670 A
`3/1999 Schuleretal. ..
`... 364/186
`Brooks, F. et al., “Project GROPE—Haptic Displays for
`oe ‘
`ty1909 Neheretal.
`0ea Scientific Visualization,” Computer Graphics, vol. 24, No.4,
`5,944,151 A
`8/1999 Jakobs el ale cesses. 1898/2671
`1990, pp. 177-185.
`5,973,670 A
`10/1999 Barber et al. ccc 345/157
`Batter, James J. et al., “Grope—1: A Computer Display to the
`5,986,643 A
`11/1999 Harvill etal. .. 345/156
`Sense of Feel,” Proc. IFIP Congress, 1971, pp. 759-763.
`
`U.S. PATENT DOCUMENTS
`
`
`
`
`
`
`
`

`

`US 6,424,333 BI
`
`Page 3
`
`Winey III, C., “Computer Simulated Visual and Tactile
`Feedback as an Aid to Manipulator and Vehicle Control,”
`Mass.Institute of Tech., Mech. Engineering, 1981, pp. 1-79.
`Burdea, G. et al., “Distributed Virtual Force Feedback,”
`IEEE Workshop on Force Display on Virtual Environments
`and its Applicationn to Robotic Teleoperation, 1993, pp.
`25-4.
`Hasser, C. et al., “Tactile Feedback with Adaptive Controller
`for a Force—-Reflecting Haptic Display,” Parts 1&2, IEEE
`0-7803-3131-1, 1996, pp. 526-533.
`Kelley, A. J. et al., “MagicMouse: Tactile and Kinesthetic
`Feedback in the Human—Computer Interface using an Elec-
`tromagnetically Actuated Input/Output Device,” Dept. of
`Elec. Eng., Univ. of Brit. Columbia, 1993, pp. 1-27.
`Wiker, Steven F. et al., “Development of Tactile Mice for
`Blind Access to Computers:
`Importance of Stimulation
`Locus, Object Size, and Vibrotactile Display Resolution,”
`Proceedings of the Human Factors Society 35th Annual
`Meeting 1991, pp. 708-712
`Gotow, J.K., et al., “Perception of Mechanical Properties at
`the Man-MachineInterface,” IEEE 1987, pp. 688-689.
`Iwata, Hiroo, “Artificial Reality with Force—feedback:
`Development of Desktop Virtual Space with Compact Mas-
`ter Manipulator,” Computer Graphics, vol. 24, No. 4, 1990,
`pp. 165-170.
`Rosenberg, Louis B. et al., “Perceptual Decomposition of
`Virtual Haptic Surfaces,” Proc. IEEE Symp. on Research
`Frontiers in Virtual Reality, Oct. 1993.
`Rosenberg, Louis B., “Virtual Haptic Overlays Enhance
`Performance in Telepresence Tasks,” Stanford Univ., Dept.
`of Mech. Eng., 1994.
`Rosenberg, Louis B., “Virtual Fixtures as Tools to Enhance
`Operator Performance in Telepresence Environments,” SPIE
`Telemanipulator Technology, 1993.
`Rosenberg, Louis B., “Perceptual Design of A Virtual Rigid
`Surface Contact,” Center for Design Research, Stanford
`University, Armstrong Laboratory, AL/CF—TR-1995-0029,
`1993, pp. 140.
`Rutherford, M. “Third Generation Digital Flight Controls,”
`CAE Electronics, Ltd., The Royal Aeronautical Society,
`1984 Spring Convention Future Applications and Prospects
`for Flight Simulation, May 9-10, 1984, paper No. 15.
`Baradat, Jean and Lacroix, Michel, “Advanced Features in
`Control Loading and Motion Systems for Simulators,”
`National Security Industrial Association 1° Interservice/
`Industry Training Equipment Conference Proceedings, Nov.
`27-29, 1981.
`
`Norlin, Ken A., “Flight Simulation Software at NASA
`Dryden Flight Research Center,” American Institute of
`Aeronautics and Astronautic’s Flight Simulation Technolo-
`gies Conference, Baltimore, MD, Aug. 7-10, 1995.
`
`Corrao, Joseph M., “Control Loading,” American Institute
`of Aeronautics and Astronautic’s Flight Simulation Update
`1987, Jan. 12-16, 1987.
`
`Corrao, J.M., “Control Loading,” American Institute of
`Aeronautics and Astronautic’s Flight Simulation Update
`1988, Jan. 11-15, 1988.
`
`“Digital Control Loading—A Modular
`P.,
`Rinaldi,
`Approach,” International Air Transport Association 6”
`Meeting of the Flight Simulator Technical Sub-Committee,
`Montreal, Jun. 1-4, 1982.
`
`Hildreth, Bruce L., Eyermann, Roger E. and Trankle, Tho-
`mas Dr., “DC Servo—Motors for High Performance High
`Reliability Control Loading in ‘light Simulators,” American
`Defense Preparedness Association 12” Interservice/Industry
`Training System Conference, Nov. 6-8, 1990.
`
`Baigrie, Stephen A., Reflectone Inc., “Electric Control
`Loading—A Low Cost, High Performance Alternative,”
`American Defense Prepardness Association 12” Interser-
`vice/Industry Training System Conference, Nov. 6-8, 1990.
`
`“Digital Control Loading”, Giel et al., Summary, Paper 1,
`Paper 2, Paper 3, International Air Transport Association,
`Seventh Flight Simulator Technical Sub—Committee Meet-
`ing, Agenda Item 10, Montreal, Sep. 17-20, 1984.
`
`Seidensticker, Steve, “Application of Microcomputersto the
`Simulator ‘Linkage’ Problem,” National Security Industrial
`Association 4Interservice/Industry ‘Training Equipment
`Conference Proceedings, Nov. 16-18, 1982.
`
`Albers, F. Gerry, “Microcomputer Base for Control Load-
`ing,” Naval Training Equipment Center 117 NTEC-Indus-
`try Conference Proceedings, NAVTRAEQUIPCENIH-306,
`Nov. 14-16, 1978.
`
`Flight Simulation, Rolfe, J.M. and Staples, K. J., eds., 1986.
`
`* cited by examiner
`
`

`

`U.S. Patent
`
`US 6,424,333 B1
`
`Jul. 23, 2002
`
`Sheet 1 of 20
`
`100.
`
`200 904
`
`200 cO3
`
`20l
`
`202
`
`FIG. 2A
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 2 of 20
`
`US 6,424,333 B1
`
`FIG. 4
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 3 of 20
`
`US 6,424,333 B1
`
`500
`
`501
`
`FIG. 5
`
`—_
`
`:
`
`>
`
`IS
`
`) )
`
`60!
`
`600
`
`FIG. 6
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 4 of 20
`
`US 6,424,333 B1
`
`700
`
`70!
`
`800
`
`}
`80!
`
`FIG. 8
`
`FIG. 7
`
`903
`
`902
`
`
`
`FIG. 9
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 5 of 20
`
`US 6,424,333 B1
`
`lO20
`
`FIG. 1OA
`
`FIG.
`
`lOB
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 6 of 20
`
`US 6,424,333 B1
`
`HO}
`
`1100
`
`HO
`
`FIG.
`
`IIA
`
`i201
`
`1200
`
`1102
`
`HOO
`103
`
`HOl
`
`toe
`
`FIG.
`
`[IB
`
`203
`
`1202
`
`1201
`
`1200
`
`1202
`
`FIG. I2A
`
`1203
`FIG. 12B
`
`

`

`US 6,424,333 B1
`
`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 7 of 20
`
`1300
`
`FIG. 14
`
`1500
`
`ISO]
`
`1502
`
`FIG. 15
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 8 of 20
`
`US 6,424,333 B1
`
`1608
`
`re 802
`1694}
`Kf
`1604 oe,
`oon)
`1602
`W=0
`We W|
`FIG.
`FIG. I6B
`
`IGA
`
`,
`
`I602
`1604 »
`
`W= Wo
`I603
`FIG. I6C
`
`
`
`FIG. 16D
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 9 of 20
`
`US 6,424,333 B1
`
`JOTNV
`
`SAIONY
`
`LNIOP
`
`SYOSN3S
`
`SO2I
`
`Z|Old
`
`cOL|
`
` K )HALNAWOO
`
`sna2021
`
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 10 of 20
`
`i802
`
`1803
`
`solof cyBla1804
`isi2 AIG
`LY)
`7
`och
`or}
`eos
`
`1809
`
`airte:iH PS1815
`
`
`IsI6
`
`yy
`Cc”
`a,
`S i805
`
`BI!
`
`FIG. ISA
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 11 of 20
`
`
`
`US 6,424,333 B1
`
`FIG. 18B
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 12 of 20
`
`US 6,424,333 B1
`
`ONISNAS
`
`AA019
`
`(O06))
`
`JWLOVLOUBIAAWALYIA
`(LO6I)LINNNVH
`
`
`
`
`(906i)
`
`
`(2061)LOVLNOODJOYOLINOW
`INIOdTWNALYIA(8061)
`
`
`
`
`
`V6lOld
`
`YOSNAS
`
`SAQ19
`
`SOVAYSILNI
`
`AWULOVLOUSIA
`
`LINA
`
`YAAINd
`
`(<O6!)
`
`YALNdGWOOD
`
`(7061)
`
`“TWWALYIA
`
`LOsfrdo
`
`(SO6I)
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 13 of 20
`
`US 6,424,333 B1
`
`3A019
`
`(O06I)
`
`FWULOVLOYSIA
`
`
`
`(o6l)LINA
`
`(2061)
`
`4.
`
`3A019
`
`YOSNAS
`
`JOVduSINI
`
`FIULOVLOUBIA
`
`TINA
`
`(c061)
`
`
`
`YALNdNOS
`LOVINODONV
`
`(vO6l)
`
`NOILVWYOANI
`
`
`
`NOILISOdLOGOY
`
`(0161)
`
`ONISNSSSVsoO
`G6!OlsYaAING
`(vI61)LOVLNODTVOISAHd=-WOISAHd
`(zI6!)NSYOSNASLOWLNOD<sS
`
`d1L090urO3L0WSYSSShes
`
`HLIMYaddlyo(E161)
`
`
`
`(1161)S\Wu
`
`4OLNIOd
`
`LOsrdo
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 14 of 20
`
`US 6,424,333 B1
`
`gaunsv3w
`
`AgogAGOg
`
`(O02)NNSOVAYSALNI
`LuvdYOSNAS
`
`FSO”™‘DIG(9002)
`
`YALNdNOD
`
`(8002)
`
`LINAAGOg
`
`(1002)WwLYVd
`
`w\)(2002)
`
`TWALYIA
`
`1004
`
`(700d)
`
`(SOOZ)
`
`LOVINOS
`
`dOLNIOd
`
`
`
`WALYIAtoargo
`
`“WALYIA
`
`JTMLOVLOYEIA
`
`STINTLoLINN
`
`
`
`FULOVLOYAIASNISNAS(6002)YOLINOW
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 15 of 20
`
`US 6,424,333 B1
`
`VIBROTACTILE
`UNIT
`(2100)
`
`S))
`
`(=
`
`SENSING
`GLOVE
`
`2101)
`
`FIG. 2IA
`
`“
`
`VIBROTACTILE
`UNIT
`(2100)
`
`( »)
`
`SENSING
`GLOVE
`(2101)
`
`FIG. 2IB
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 16 of 20
`
`US 6,424,333 B1
`
`GaPy
`aD
`ma
`VIBROTACTILE
`(221
`
`CLARINET
`
`(2200)
`
`DRUMS
`
`(2201)
`
`SECOND
`
`(2202)
`
`TRUMPET
`(2203)
`
`GUITAR
`(2210)
`
` :
`
`VIOLIN
`mo
`oo
`N
`(2207) OT
`
`FLUTE
`(2206)
`
`FRENCH
`HORN
`(2209)
`
`BASS
`(2205)
`
`|
`
`CELLO
`(2204)
`
`SAXOPHONE
`(2208)
`
`

`

`PROPAGATION
`
`ACTIVATION SIGNAL
`
`2302
`
`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 17 of 20
`
`US 6,424,333 B1
`
`2303
`
`NN
`\\
`
`
`
`FIG. 23
`
`2400
`
`sao,
`
`3A05
`
`2401
`
`( ,
`
`FIG. 24A
`
`FIG. 24B
`
`

`

`Jul. 23, 2002
`
`Sheet 18 of 20
`
`US 6,424,333 B1
`
`
`
`U.S. Patent
`
`2502
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 19 of 20
`
`US 6,424,333 B1
`
`2600
` 2601 eos
`
`C602 «ae
`
`FIG. 26B
`
`2604
`
`<>
`
`2600
`
`260!
`
`2606
`
`oe 2603
`FIG.26C
`
`

`

`U.S. Patent
`
`Jul. 23, 2002
`
`Sheet 20 of 20
`
`US 6,424,333 B1
`
`NOLLVYSIA
`
`FILLOVL
`
`LLZz
`
`piizGlL2CLL?
`
`1uVdAGOE¢12ZIWNOIS
`
`
`ONISNSSwAATNGYOSSH00Ud=!aIVIS
`
`
`O1lZ(izzz).TWNOIS|!|TALIA
`
`
`
`26‘SlsALVISWOISAHdWOrSAHd
`WNOIS|ONLLVALLOW
`TWNOIS7TOYLNOD|£022>f
`ffeBOLE
`4008¥||“aaunsvan—4OUINOWYo!|TWALYIAUe
`JEvoPoee~_”
`
`[7a4---~~~,lavdAgog
`‘TWWNOISAIVISds3ynsvan
`
`
`!
`
`WOISAHd
`
`EZLZ~,
`
`
`
`

`

`US 6,424,333 Bl
`
`1
`TACTILE FEEDBACK MAN-MACHINE
`INTERFACE DEVICE
`
`This is a continuation application of prior U.S. applica-
`tion Ser. No. 09/561,782,filed on May 1, 2000 now US. Pat.
`No. 6,275,213, in the name of Marc Tremblay, et al., which
`is a continuation of prior U.S. patent application Ser. No.
`09/066,608, filed on Apr. 24, 1998 now U.S. Pat. No.
`6,088,017, which is a continuation of U.S. patent application
`Ser. No. 08/565,102,filed Nov. 30, 1995, abandoned;andall
`ot which are incorporated herein by reference.
`TECHNICAL FIELD
`This invention relates to a man-machine interface and in
`
`particular to an interface that provides tactile sensation to a
`user.
`
`10
`
`15
`
`BACKGROUND OF THE INVENTION
`
`2
`SUMMARY OF THE INVENTION
`
`An object of the invention is a man-machine interface
`which may be employed in such areas as interactive com-
`puter applications, telerobotics, gesture recognition, music
`generation, entertainment, medical applications andthelike.
`Another object of the invention is a mass which is moved by
`a “mass-moving actuator” which generates a vibration that
`a uscr can fecl. Yet another object of the invention is the
`generation of an activating signal to produce the vibrations
`either as a result of the user’s state or as a result of
`
`environmental conditions, whether virtual or physical. Stull
`another object of the invention is vibrating the bone struc-
`ture of
`a sensing body part, as well as skin
`mechanoreceptors,
`to provide feedback. Yet still another
`object of the invention is the complex actuation of vibratory
`devices.
`
`The tactile sensation that a user feels is generated by a
`vibrotactile unit mounted on, or in functional relation to, a
`sensing body part of a user by a fastening means. In one
`embodiment, the vibrotactile device comprises a mass con-
`nected eccentrically to a mass-movingactuatorshaft(.e. the
`center of mass of the massis offset from the axis of rotation).
`Energizing the mass-moving actuator causes the shaft to
`turn, which rotates the eccentric mass. This rotating mass
`causes a corresponding rotating force vector. A rapidly
`rotating force vectorfeels to the user as a vibration. Aslowly
`rotating force vector feels like a series of individual
`impulses. For a small numberofrapid rotations, the rotating
`force vectorfeels like a single impulse. We will use the term
`“vibration” to denote a change in force vector(i.¢c., direction
`or magnitude). Examples of vibrations include, but are not
`limited to a single impulse, a sinusoidal force magnitude,
`and other functions of the force vector. We use the term
`
`“tactile sensation”to refer to the feeling perceived by a user
`whentheir sensing body part experiences vibrations induced
`by a vibrotacule unit.
`A signal processor interprets a state signal and produces
`an activating signal to drive the mass-moving actuator. The
`variable components of the state signal may be physical
`(e.g., measured), or virtual (e.g. simulated, or internally
`generated); they may vary with time(e.g., the state variables
`may represent processes); and they may be integer-valued
`(e.g., binary or discrete) or real-valued (e.g., continuous).
`The signal processor may or may not comprise a computer
`which interprets and further processes the state signal. The
`signal processor comprises a signal driver which produces
`an activating signal supplying power to, or controlling the
`powerdrawnby, the vibrotactile unit. The power maybe, but
`is notrestricted to, electric, pneumatic, hydraulic, and com-
`bustive types. The driver may be, but is not restricted to, an
`electric motor controller comprising a current amp and
`sensor for closed loop control, a flow valve controlling the
`amountof a pressurized fluid or gas, a flow valve controlling
`the amountof fuel to a combustion engine and the like. The
`details of such a signal processor and mass-moving actuator
`are common knowledge to someone skilled in the art.
`The state signal may be generated in responseto a variety
`of conditions. In one embodiment, one or more sensors
`measuring physical conditions of the user and/or the user’s
`environment may generale one or more components of a
`physical state signal. In another embodiment, a computer
`simulation may determine the one or more components of a
`virtual state signal from a simulated (e.g., virtual) state or
`condition. The virtual state may optionally be influenced by
`a physical state. The virtual state includes anything that a
`computer or timing system can generate including, but not
`
`Virtual reality (VR) is an immersive environment which
`is created by a computer and with which users have real-
`time, multisensorial interactions. Typically, these interac-
`tions involve someorall of the human senses through either
`visual feedback, sound, force and tactile feedback (ie.
`reflection), smell and even taste. The key to immersive
`realism is the capacity of the user to use his/her hand to 5
`interactively manipulate virtual objects. Unfortunately, the
`majority of existing commercial virtual reality systems use
`hand-sensing devices that provide no haptic feedback.
`Nevertheless, someefforts have been madeto provide means
`for presenting force and tactile information to the user’s
`hand. By force information, it is meant the application of a
`set force ta a selected part of the hand, for example,a finger.
`By tactile information,
`it
`is meant
`the application of a
`stimuli, e.g., a vibration, to a selected part of the hand,e.g.,
`a fingertip pad. This stimulus, could simulate surface texture
`or dynamic conditions at the contact, for example. A few
`examples of existing force reflecting devices are the EXOS
`SAFiRE™, the Master I] Hand Master device at Rutgers
`university, the PERCRO Force-Reflecting Hand Master and
`the Sarcos TOPS Force-Reflecting Hand Master. Some
`tactile feedback devices that have been developed include
`the PERCRO Position-Sensing and Tactile Feedback Hand
`Master and the EXOS TouchMaster™.
`
`30
`
`35
`
`40
`
`Virtual reality is not the only field where it is desirable to
`feed back force and tactile information to a human user/
`operator. Another commonarea is telerobotics. Someof the
`devices mentioned aboveare also often used as telerobotics
`interfaces. Some examples in the literature of feedback
`devices designed more specifically for telerobotics include
`the tactile shape sensing and display system developed by
`Kontariniset al., the voice-coil based tactile feedback device
`used by Patrick et al. and the pin-basedtactile display array
`developed by Kaczmarek and Bach-y-rita. Other applica-
`tions for a vibrotactile unit of the subject invention include,
`but are not limited to, gesture recognition, music generation,
`entertainment and medical applications.
`In an ideal case, it would be desirable to provide full force
`and tactile feedback to a user to make the virtual reality or
`telerobotic experience as realistic as possible. Unfortunately,
`most force feedback devices are cumbersome, heavy, expen-
`sive and difficult to put on and remove. Manyofthetactile
`feedback solutions are also cumbersome, complex andfrag-
`ile. Additionally, some of the tactile feedback devices
`described in the literature, such as small voice coils mounted
`to directly contact the skin, tend to numbthe skin after only
`a few seconds of operation and then become ineffective as
`feedback devices.
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`US 6,424,333 Bl
`
`3
`restrictedto, a fixed time froma previous event; the position,
`velocity, acceleration (or other dynamic quantity) of one or
`more virtual objects in a simulation; the collision of two
`virtual objects in a simulation; the start or finishing of a
`computer job or process; the setting of a flag by another
`process or simulation; combinations of situations; and the
`like. The virtual state signal is a machine-readable measure-
`ment of the virtual state variables.
`
`The physical state signal is measured from physical state
`variables. These variables have relevance to the physical
`state of a body part of the user or the user’s physical
`cnvironment. The physical state variables includes any mca-
`surable parameter in the environment or any measurable
`parameter relating to a bodypart of the user. Some examples
`of measurable physical parameters in an environment
`include butare notrestricted to, the state of a body part, the
`position of objects in the environment, the amountof energy
`imparted to an object in the environment, the existence of an
`object or objects in the environment, the chemical state of an
`object, the temperature in the environment, and the like. The
`state of a body part may include the physical position,
`velocity, or acceleration of the bodypart relative to another
`bodypartor relative to a point in the environment. The state
`of a body part may also include any bodily function, where
`the measured state signal may include the output from an
`electroencephalograph (EEG), electrocardiograph (ECG),
`electromyograph (EMG), electrooptigraph (EOG) or eye-
`gaze sensor, and sensors which measure joint angle, heart
`rate, dermal or subdermal
`temperature, blood pressure,
`blood oxygen content (or any measurable blood chemical),
`digestive action, stress level, voice activation or voice
`recognition, and the like. The user’s voice may constitute a
`measured physical state variable, where his spoken words
`are sensed and/or recognized to generate a corresponding
`activating signal. The physical state signal is a machine-
`readable measurement of the physical state variables.
`The state signal is presented to the signal processor which
`interprets the state, and then determines how and whento
`activate the vibrotactile units accordingly. The signal pro-
`cessor produces an activating signal which may be in
`response to an event
`it interprets from the state signal.
`Examplesof events include contact, gestures, spoken words,
`onset of panic or unconsciousness, and the like. The inter-
`pretation of the state signal may or may not be a binary
`event, i.e. the simple changing of state between two values.
`An example of a binary event is contact vs. non-contact
`between two virtual or real objects. The process of inter-
`preting may include any general function of state variable
`components. The interpretation function may produce an
`output control value which is integer or real-valued. A
`non-binary-valued interpretation output typically relates to
`the signal processor producing a non-binary activation sig-
`nal.
`
`By varying the functional form of the activation signal,
`the type of feedback that the vibrotactile device generates
`may also be varied. The device may generate a complex
`tactile sensation, which is defined to be a non-binary signal
`from a single or multiple vibrotactile units. Examples of
`complextactile sensations include (1) varying the amplitude
`of vibration with a profile which is non-uniform overtime;
`(2) varying the frequency of vibration; (3) varying the
`duration of impulses; (4) varying the combination of ampli-
`tude and frequency; (5) vibrating two or more vibrotactile
`units with a uniform or non-uniform amplitude profile; (6)
`sequencing multiple vibrotactile units with different ampli-
`tude or frequencyprofiles; and the like.
`The frequency and amplitude of the vibration or impulse
`may be changed by modifying the activating signal to the
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`mass-moving, actuator. The frequency and amplitude may
`also be controlled by increasing the mass or by changing the
`radius of gyration (e.g. changing its eccentricity). For
`example, the mass may be changed by pumpingfluid into an
`eccentrically rotating container. The sense of frequency that
`the user perceives may be changed independently of the
`amplitude by modulating the powerto the vibrotactile unit
`at a variable frequency. This technique is called amplitude
`modulation, which is common knowledgeto those skilled in
`the art. This change in frequency and amplitude may be used
`to convey complex, compoundor other forms of information
`to the user.
`
`Sensors may be mounted on the vibrotactile unit or the
`sensing bodypart to determine the frequency and amplitude
`of vibration sensed by the user. A feedback control loop may
`be added which usesthis information to more tightly control
`the frequency and amplitude, or to reach peak efficiency at
`the resonant frequency of the collective vibrating device-
`body system.
`Examples of a sensing body part on which the vibrotactile
`unit may be mountcd, or
`in functional relation to the
`vibrotactile unit, include, but are not limited to: the distal
`part of a digit,
`the dorsal (back) side of a phalanx or
`metacarpus, palm, forearm, humerus, underarm, shoulder,
`back, chest, nipples, abdomen, head, nose, chin, groin,
`genitals, thigh, calf, shin, foot, toes, and the like. A plurality
`of vibrotactile units may be disposed on or near different
`sensing body parts, and may be activated in unison or
`independently.
`Each vibrotactile unit may be affixed to the body by a
`fastening means. The fastening means is defined to be the
`meansof attaching the vibrotactile unit to a sensing body
`part, transmitting (and possibly modifying) the vibrations
`created by the vibrotactile unit. This means maybe one that
`is flexible such as a strap made of cloth or soft polymer, or
`rigid, such as metal or hard polymer which grabs or pinches
`the flesh, skin or hair. The fastening means mayalso include
`gluing or taping to the skin or hair, or tying with a string or
`rope around a limb,or attaching to clothes with VelcroTM
`or similarly functional means. A vibrotactile unit may also
`be attached to another structure whichis then attached to the
`body part with the same means just mentioned. The vibra-
`tions generated by the actuator may be transmitted to the
`sensing body part by the structure (rigid or non-rigid), or
`through a linkage transmission or a fluid transmission.
`The eccentric mass need not be mounted directly onto a
`motor shaft. A mechanical transmission mayrotate the mass
`on a different shaft than the motor shaft. The mass-moving
`actuator rotates this shaft. Fluids such as air and liquids may
`also transmit the motion from a power sourceto the rotating
`eccentric mass. Changing magnetic fields may also be
`employed to induce vibration of a ferrous mass.
`As previously mentioned, state signals may relate to a
`physicalor virtual state. When the state represents a physical
`condition, the subject invention includes a state measure-
`ment sensor which producesa state signal. This state mea-
`surement sensor may measure somepropertyof the sensing
`body part. Recall that the body part associated with receiv-
`ing the vibrotactile stimulation is called the sensing body
`part, the body part associated with producing the activating
`signalis called the measured body part. The signal processor
`may receive signals from this sensor such as a tactile,
`position, bend, velocity, acceleration or temperature sensor
`and generate an activating signal. In this way, the user may
`receive feedback based on his actions or physical state. For
`example, the vibrotactile device may be used to train the
`
`

`

`US 6,424,333 Bl
`
`5
`the
`user to do some physical motion task. In this case,
`position or motion of the body part which is to do the motion
`task is