(15
`United States Patent
`6,088,017
`Jul. 11, 2000
`[45] Date of Patent:
`Tremblay et al.
`
`[11] Patent Number:
`
`US006088017A
`
`[54] TACTILE FEEDBACK MAN-MACHINE
`INTERFACE DEVICE
`
`[75]
`
`Inventors: Mark R. Tremblay, Mountain View;
`Mark H. Yim, Palo Alto, both of Calif.
`
`[73] Assignee: Virtual Technologics, Inc., Palo Alto,
`Calif.
`
`WO 96/09695
`
`3/1996 WIPO .
`
`OTHER PUBLICATIONS
`
`Hasser, C.J., “Force-refelecting Anthropomorphic Hand
`Masters,” Interim Report, Air Force Material Command,
`Wright-Patterson Air Force Base, Jul. 1995.
`
`[21] Appl. No.: 09/066,608
`
`[22]
`
`Filed:
`
`Apr. 24, 1998
`
`Primary Examiner—Regina Liang,
`Attorney, Agent, or Firm—Flehr Hohbach Test Albritton &
`Herbert LLP
`
`Related U.S. Application Data
`
`[57]
`
`ABSTRACT
`
`[63] Continuation of application No. 08/565,102, Nov. 30, 1995,
`abandoned.
`
`Tint. C07ccc eeeeeneeeeecceensenmameeeeee G09G 5/00
`[51]
`[52] US. Ch ow.
`345/156; 414/5
`
`[58] Field of Search 2...eee 345/156, 158,
`345/157, 145, 146; 414/2, 5, 1, 3, 4, 6,
`7; 901/32-34
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`«» 128/774
`11/1983 Zarudiansky..
`4,414,984
`
`«. 345/156
`3/1988 McRaeetal. ....
`4,731,603
`
`364/513.5
`9/1991 Krameret al.
`5,047,952
`2/1993 Kramer.........
`-- 364/806
`5,184,319
`
`2/1995 Franklin et al...ee 434/114
`5,388,992
`FOREIGN PATENT DOCUMENTS
`
`Aman-machine interface which providestactile feedback to
`various sensing bodypartsis disclosed. The device employs
`one or more vibrotactile units, where each unit comprises a
`mass and a mass-moving actuator. As the mass is accelerated
`by the mass-moving actuator,
`the entire vibrotactile unit
`vibrates. Thus, the vibrotactile unit transmits a vibratory
`stimulus to the sensing body part to whichit is affixed. The
`vibrotactile unit may be used in conjunction with a spatial
`placement sensing device which measures the spatial place-
`ment of a measured bodypart. A computing device uses the
`spatial placementof the measured bodypart to determine the
`desired vibratory stimulus ta be provided by the vibrotactile
`unit. In this manner, the computing device may control the
`level of vibratory feedback perceived by the corresponding
`sensing body part in response to the motion of the measured
`body part. The sensing bodypart and the measured bodypart
`may be separate or the same bodypart.
`
`626 634 A2
`
`11/1994 European Pat. Off.
`
`.
`
`24 Claims, 20 Drawing Sheets
`
`1707
`
`BUS
`
`
`
`COMPUTER
`
`
`
`ANALOG
`JOINT
`ANGLE
`SENSORS
`
`
`
`702
`
`(705
`
`APPLE 1013
`
`APPLE 1013
`
`1
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 1 of 20
`
`6,088,017
`
`{@}s)
`
`lOO
`
`
`
`“ei
`Oe
`FIG.
`
`IB
`
`104
`
`200 294 «200
`
`2Ol
`
`202
`
`FIG. 2A
`
`203
`
`200
`
`FIG. 2B
`
`2
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 2 of 20
`
`6,088,017
`
`3
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 3 of 20
`
`6,088,017
`
`COTTA
`L\=>
`Se 60!
`
`—>
`
`IN
`
`/ )
`
`600
`
`FIG. 6
`
`iN
`
`4
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 4 of 20
`
`6,088,017
`
`700
`
`7Ol
`
`FIG. 7
`
`902
`
`800
`
`903
`
`FIG. 9
`
`FIG. 8
`
`5
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 5 of 20
`
`6,088,017
`
`i020
`
`FIG.
`
`[OA
`
`FIG. 1OB
`
`6
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 6 of 20
`
`6,088,017
`
`HO}
`
`lOO
`
`103
`
`FIG. HA
`
`1201
`
`1200
`
`1102
`
`HOS
`
`oe
`
`FIG.
`
`IIB
`
`1203
`
`1202
`
`120]
`
`{ZOO
`
`i202
`
`FIG. I2A
`
`1203
`FIG. [2B
`
`7
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 7 of 20
`
`6,088,017
`
`FIG. 14
`
`I500
`
`ISO}
`
`1502
`
`FIG. 15
`
`8
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 8 of 20
`
`6,088,017
`
`M
`
`KI
`i602 my
`W=O
`FIG.
`
`1603
`
`1604
`
`I16A
`
`M
`
`i604
`
`1603
`
`1602
`
`(
`W= Wa
`M
`W=W,
`FIG. I6B FIG. I6C
`
`i602
`
`1604
`
`1603
`
`
`
`FIG. 16D
`
`9
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 9 of 20
`
`6,088,017
`
`co
`
`
`ANALOGJOINTANGLESENSORS
`
`
`©&
`
`1701
`
`1705
`
`7 F
`
`IG.
`
`1702
`
`BUS
`
`\707
`
`
`
` COMPUTER
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 10 of 20
`
`6,088,017
`
` FIG. [8A
`
`11
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 11 of 20
`
`6,088,017
`
`
`
`12
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 12 of 20
`
`6,088,017
`
`ONISNSS
`
`3A019
`
`(O06))
`
`V6I‘|4(£O06))
`
`YSAIYO
`
`YOSNAS
`
`3A019
`
`SOVAYSALNI
`
`AVLOVLOYGIA
`
`-LINA
`
`YALNdWOOD
`
`(7061)
`
`
`
`SULOVLOYBIATWWALYIA
`
`
`
`(2061)LOVLNOD34OYOLINOW
`
`
`
`
`
`LNIOdWWNALYIA(BO6))
`
`
`
`(1061)LINN
`
`ant
`
`TWOLYIA
`
`193rgo
`
`(SO6I)
`
`13
`
`13
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 13 of 20
`
`6,088,017
`
`
`LOvINOD_koSS(O61)LINNHLIMYaddlyo(S161)
`
`
`
`
`FULOVLONBIA193¢°d0
`
`4OLNIOd
`
`
`
`
`
`
`
`(VI6I)LOVLNODTVOISAHdIWOISAHd
`
`(2161)\\INHOSNAS
`
`\,
`
`ONISNASQ\SLOWSHSs6zSASS
`(0061)31L0G0uES,
`
`(0061)Wa
`
`(€06!)
`
`
`
`uaAluGg6lOl4TINA(0161)
`
`
`
`
`
`FULIVLOUBIA(boel)NOLLVWHOSNI
`
`YALNdNOD
`LOVLNODONY
`
`
`
`
`
`JOVAYSILNINOILISOdLOGON
`
`3A0194(206I)
`
`HOSN3Sfe
`
`ob
`
`14
`
`Gel)S\
`
`CO
`
`14
`
`
`
`
`

`

`
`
`
`
`
`
`AULOVLOYEIAONISNAS(6002)YOLINOW
`
`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 14 of 20
`
`6,088,017
`
`
`
`(£002)SJOVAYMALNI
`
`-—S\Oc9IS4(9002)
`
`AGOAGOg(8002)
`1uvdMOSN3S
`
`G3YyNSV3SWN
`
`Y3SLNdNWOD
`
`LINAAQOg
`
`(i002)
`
`e\)(2002)WwLYVd
`
`(SOOZ)
`
`LOVINOD
`
`dOLNlOd
`
`SMLOVLOYGIA
`
`TWWALYIA
`
`LOod
`
`(7002)
`
`
`
`VWOALYIA1o3rgo
`
`TWOALYIA
`
`15
`
`LINA
`
`
`
`used—v2-
`
`15
`
`
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 15 of 20
`
`6,088,017
`
`VIBROTACTILE
`UNIT
`(2100)
`
`1)
`
`SENSING
`GLOVE
`(2101)
`
`FIG. 21A
`
`A
`
`VIBROTACTILE
`UNIT
`(2100)
`
`( »)
`
`SENSING
`GLOVE
`(2101)
`
`FIG. 2IB
`
`16
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 16 of 20
`
`6,088,017
`
`(2200)
`
`CLARINET
`
`FIRST
`VIOLIN
`(2207)
`
`SECOND
`VIOLIN
`(2202)
`
`TRUMPET
`(2203)
`
`VIBROTACTILE
`UNIT
`
`(2211
`DRUMS
`(2201)
`
`
`
`
`
`
`
`
`
`
`FLUTE
`(2206)
`
`GUITAR
`(2210)
`
`
`CELLO
`FRENCH
`(2204)
`HORN
`(2209)
`
`BASS
`(2205)
`
`SAXOPHONE
`(2208)
`
`17
`
`17
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 17 of 20
`
`6,088,017
`
`
`
`2303
`
`
`
`230)
`
`ACTIVATION SIGNAL
`
`PROPAGATIONLL C'S
`
`oD
`
`—
`
`i
`
`
`
`FIG. 23
`
`2400
`
`2401
`
`2400
`
`2401
`
`((
`
`))
`
`FIG. 24A
`
`FIG. 24B
`
`18
`
`18
`
`

`

`Jul. 11, 2000
`
`Sheet 18 of 20
`
`U.S. Patent
`
`6,088,017
`
`AG on
`
`250!
`
`-
`
`2502
`
`~2503
`
`FIG. 25B
`
`19
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 19 of 20
`
`6,088,017
`
`2600
` 260!
`
`pO2603
`
`2602 «>
`
`2601
`
`2606
`
`FIG. 26B
`
`2604
`
`>
`
`2600
`
`oe 2603
`FIG.26C
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Jul. 11, 2000
`
`Sheet 20 of 20
`
`6,088,017
`
`
`
`LuVdAQOE¢L/Z
`viézGide=Gide
`NOLLVUaIA
`ONLLVALLOV
`£2‘Old
`
`JOYLNOD
`
`YOSS3A00Ud
`||||||(4022)
`YaLNdNOD©
`
`TWNOIS
`
`“WALLYIA
`
`SLVLS
`
`£OL2
`
`L
`
`
`|tevdA008—Nounony||
`qaynsvan/TWALUIA“
`
`“
`
`ONISNAS
`
`OLZ?e
`
`YSAING
`
`IWNOIS
`
`(1227)
`
`FILLOVL
`
`Lb2z
`
`WNOIS
`
`!ETLZ~.
`
`
`|;alvisqWoiShHd-VOISAHd
`
`|)WWOISAHdl||QLZZI|!|||PLL1
`potA7!|luvdAGOg|i} ‘TWNOIS
`aivisCSHOSVAN
`
`“WNOIS
`PSS)
`NOLLVINWIS
`
`ZZLZ
`
`21
`
`21
`
`
`
`
`
`
`

`

`6,088,017
`
`1
`TACTILE FEEDBACK MAN-MACHINE
`INTERFACE DEVICE
`
`Thisis a continuation of application Ser. No. 08/565,102,
`filed Nov. 30, 1995 now abandoned.
`
`RELEVANT LITERATURE
`
`U.S. PATENT DOCUMENTS
`
`US. Pat. No. 4,414,984 November 1983 Zarudiansky
`US. Pat. No. 5,184,319 February 1990 Kramer
`
`OTHER PUBLICATIONS
`
`C. J. Hasser, “Force-reflecting Anthropomorphic Hand
`masters,” Interim Report, Air Force Materiel Command,
`Wright-Patterson Air Force Base, July 1995.
`K. Kaczmarek and P. Bach-y-rita, “Tactile Displays,” in
`Advanced Interface Design and Virtual Environments, W.
`Barfield and T. Furness III Editors, Oxford University Press,
`1993.
`
`D. A. Kontarinis and R. D. Howe, “Display of High-
`frequency Tactile Information to Teleoperators,” Proceed-
`ings of the SPIE—tIhe International Society for Optical
`Engineering, 1993, Vol. 2057, pp. 40-50.
`N. J. M. Patrick, T. B. Sheridan, M. J. Massimino and B.
`A.Marcus,“Design and Testing of a Non-reactive, Fingertip
`Tactile Display for Interaction with Remote Environments,”
`Proceedings of the SPIE—tIhe International Society for
`Optical Engineering, 1991, Vol. 1387, pp. 215-222.
`
`TECHNICAL FIELD
`
`10
`
`15
`
`30
`
`This invention relates to a man-machine interface and in
`
`particular to an interface that provides tactile sensation to a
`user.
`
`35
`
`BACKGROUND OF THE INVENTION
`
`Virtual reality (VR) is an immersive environment which
`is created by a computer and with which users havereal-
`time, multisensorial interactions. Typically, these interac-
`tions involve someorall of the human senses through either
`visual feedback, sound, force and tactile feedback (ie.
`reflection), smell and even taste. The key to immersive
`realism is the capacity of the user to use his/her hand to
`interactively manipulate virtual objects. Unfortunately, the
`majority of existing commercial virtual reality systems use
`hand-sensing devices that provide no haptic feedback.
`Nevertheless, someefforts have been madeto provide means
`for presenting force and tactile information to the user’s
`hand. By force information, it is meant the application of a
`set force to a selected part of the hand, for example,a finger.
`By tactile information,
`it
`is meant
`the application of a
`stimuli, e.g., a vibration, to a selected part of the hand,e.g.,
`a fingertip pad. This stimulus, could simulate surface texture
`or dynamic conditions at the contact, for example. A few
`examples of existing force reflecting devices are the EXOS
`SAFiRE™, the Mastcr HT Hand Master device at Rutgers
`university, the PERCRO Force-Reflecting Hand Master and
`the Sarcos TOPS Force-Reflecting Hand Master. Some
`tactile feedback devices that have been developed include
`the PERCRO Position-Sensing and Tactile Feedback Hand
`Master and the EXOS TouchMaster™.
`
`Virtual reality is not the only field where it is desirable to
`feed back force and tactile information to a human user/
`operator. Another commonarea is telerobotics. Someof the
`devices mentioned above are also often used as telerobotics
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`interfaces. Some examples in the literature of feedback
`devices designed more specifically for telerobotics include
`the tactile shape sensing and display system developed by
`Kontarinis et al., the voice-coil based tactile feedback device
`used byPatrick et al. and the pin-basedtactile display array
`developed by Kaczmarek and Bach-y-rita. Other applica-
`tions for a vibrotactile unit of the subject invention include,
`but are not limited to, gesture recognition, music generation,
`entertainment and medical applications.
`In an ideal case, it would be desirable to provide full force
`and tactile feedback to a user to make the virtual reality or
`telerobotic experienceas realistic as possible. Unfortunately,
`most force feedback devices are cumbersome, heavy, expen-
`sive and difficult to put on and remove. Manyof thetactile
`feedback solutions are also cumbersome, complex and frag-
`ile. Additionally, some of the tactile feedback devices
`described in the literature, such as small voice coils mounted
`to directly contact the skin, tend to numbthe skin after only
`a few seconds of operation and then become ineffective as
`feedback devices.
`
`SUMMARY OF THE INVENTION
`
`An object of the invention is a man-machine interface
`which may be employed in such areas as interactive com-
`puter applications, telerobotics, gesture recognition, music
`generation, entertainment, medical applicationsandthelike.
`Another object of the invention is a mass which is moved by
`a “mass-moving actuator” which generates a vibration that
`a user can feel. Yet another object of the invention is the
`generation of an activating signal to produce the vibrations
`either as a result of the user’s state or as a result of
`environmental conditions, whether virtual or physical. Still
`another object of the invention is vibrating the bone struc-
`ture of
`a sensing body part, as well as skin
`mechanoreceptors,
`to provide feedback. Yet still another
`object of the inventionis the complex actuation of vibratory
`devices.
`
`The tactile sensation that a user feels is generated by a
`vibrotactile unit mounted on, or in functional relation to, a
`sensing, body part of a user by a fastening means. In one
`embodiment, the vibrotactile device comprises a mass con-
`nected eccentrically to a mass-movingactuator shaft(i.e. the
`center of mass of the massis offset from the axis of rotation).
`Energizing the mass-moving actuator causes the shaft to
`turn, which rotates the eccentric mass. This rotating mass
`causes a corresponding rotating force vector. A rapidly
`rotating force vectorfeels to the user as a vibration. Aslowly
`rotating force vector feels like a series of individual
`impulses. For a small numberofrapid rotations, the rotating
`force vectorfeels like a single impulse. We will use the term
`“vibration” to denote a change in force vector (i¢., direction
`or magnitude). Examples of vibrations include, but are not
`limited to a single impulse, a sinusoidal force magnitude,
`and other functions of the force vector. We use the term
`
`“tactile sensation”to refer to the feeling perceived by a user
`whentheir sensing body part experiences vibrations induced
`by a vibrotactile unit.
`A signal processor interprets a state signal and produces
`an activating signal to drive the mass-moving actuator. The
`variable components of the state signal may be physical
`(e.g., measured), or virtual (e.g. simulated, or internally
`generated); they may vary with time(e.g., the state variables
`may represent processes); and they may be integer-valued
`(e.g., binary or discrete) or real-valued (e.g., continuous).
`The signal processor may or may not comprise a computer
`which interprets and further processes the state signal. The
`
`22
`
`22
`
`

`

`6,088,017
`
`
`
`3
`signal processor comprises a signal driver which produces
`an activating signal supplying power to, or controlling the
`powerdrawnby,the vibrotactile unit. The power maybe,but
`is notrestricted to, electric, pneumatic, hydraulic, and com-
`bustive types. The driver may be, but is not restricted to, an
`electric motor controller comprising a current amp and
`sensor for closed loop control, a flow valve controlling the
`amountof a pressurized fluid or gas, a flow valve controlling
`the amountof fuel to a combustion engine andthe like. The
`details of such a signal processor and mass-moving actuator
`are common knowledge to someone skilled in theart.
`The state signal may be generated in response to a variety
`of conditions. In one embodiment, one or more sensors
`measuring physical conditions of the user and/or the user’s
`environment may generate one or more components of a
`physical state signal. In another embodiment, a computer
`simulation may determine the one or more components of a
`virtual state signal from a simulated (e.g., virtual) state or
`condition. The virtual state may optionally be influenced by
`a physical state. The virtual state includes anything that a
`computer or timing system can generate including, but not
`restricted to, a fixed time from a previous event; the position,
`velocity, acceleration (or other dynamic quantity) of one or
`more virlual objects in a simulation; the collision of two
`virtual objects in a simulation; the start or finishing of a
`computer job or process; the setting of a flag by another
`process or simulation; combinations of situations; and the
`like. ‘The virtual state signal is a machine-readable measure-
`mentof the virtual state variables.
`
`The physical state signal is measured from physical state
`variables. These variables have relevance to the physical
`state of a body part of the user or the user’s physical
`environment. The physical state variables includes any mea-
`surable parameter in the environment or any measurable
`parameter relating to a bodypart of the user. Some examples
`of measurable physical parameters in an environment
`include butare notrestricted to, the state of a body part, the
`position of objects in the environment, the amountof energy
`imparted to an object in the environment, the existence of an
`object or objects in the environment, the chemical state of an
`object, the temperature in the environment, and the like. The
`state of a body part may include the physical position,
`velocity, or acceleration of the bodypart relative to another
`bodypartor relative to a point in the environment. Thestate
`of a bodypart may also include any bodily function, where
`the measured state signal may include the output from an
`electroencephalograph (EEG), electrocardiograph (ECG),
`electromyograph (EMG), electrooptigraph (EOG) or eye-
`gaze sensor, and sensors which measure joint angle, heart
`rate, dermal or subdermal
`temperature, blood pressure,
`blood oxygen content (or any measurable blood chemical),
`digestive action, stress level, voice activation or voice
`recognition, and the like. The user’s voice may constitute a
`measured physical state variable, where his spoken words
`are sensed and/or recognized to generate a corresponding
`activating signal. The physical state signal is a machine-
`readable measurement of the physical state variables.
`The state signal is presented to the signal processor which
`interprets the state, and then determines how and whento
`activate the vibrotactile units accordingly. The signal pro-
`cessor produces an activating signal which may be in
`response to an event
`it interprets from the state signal.
`Examples of events include contact, gestures, spoken words,
`onset of panic or unconsciousness, and the like. The inter-
`pretation of the state signal may or may not be a binary
`event, i.e. the simple changing of state between two values.
`An example of a binary event is contact vs. non-contact
`
`4
`between two virtual or real objects. The process of inter-
`preting may include any general function of state variable
`components. The interpretation function may produce an
`output control value which is integer or real-valued. A
`non-binary-valued interpretation output typically relates to
`the signal processor producing a non-binary activation sig-
`nal.
`
`By varying the functional form of the activation signal,
`the type of feedback that the vibrotactile device generates
`may also be varied. The device may generate a complex
`tactile sensation, which is defined to be a non-binary signal
`from a single or multiple vibrotactile units. Examples of
`complextactile sensations include (1) varying the amplitude
`of vibration with a profile which is non-uniform over time;
`(2) varying the frequency of vibration; (3) varying the
`duration of impulses; (4) varying the combination of ampli-
`tude and frequency; (5) vibrating two or more vibrotactile
`units with a uniform or non-uniform amplitude profile; (6)
`sequencing multiple vibrotactile units with different ampli-
`tude or frequency profiles; and the like.
`The frequency and amplitude of the vibration or impulse
`may be changed by modifying the activating signal to the
`mass-moving actuator. The frequency and amplitude may
`also be controlled by increasing the mass or by changing the
`radius of gyration (e.g. changing its eccentricity). For
`example, the mass may be changed by pumping fluid into an
`eccentrically rotating container. The sense of frequency that
`the user perceives may be changed independently of the
`amplitude by modulating the powerto the vibrotactile unit
`at a variable frequency. This technique is called amplitude
`modulation, which is common knowledgeto those skilled in
`the art. This change in frequency and amplitude may be used
`to convey complex, compound or other forms of information
`to the user.
`
`Sensors may be mounted on the vibrotactile unit or the
`sensing body part to determine the frequency and amplitude
`of vibration sensed by the user. A feedback control loop may
`be added which usesthis information to more tightly control
`the frequency and amplitude, or to reach peak efficiency at
`the resonant frequency of the collective vibrating device-
`body system.
`Examples of a sensing body part on which the vibrotactile
`unit may be mounted, or
`in functional relation to the
`vibrotactile unit, include, but are not limited to: the distal
`part of a digit,
`the dorsal (back) side of a phalanx or
`metacarpus, palm, forearm, humerus, underarm, shoulder,
`back, chest, nipples, abdomen, head, nose, chin, groin,
`genitals, thigh, calf, shin, foot, toes, and the like. A plurality
`of vibrotactile units may be disposed on or near different
`sensing body parts, and may be activated in unison or
`independently.
`Each vibrotactile unit may be affixed to the body by a
`fastening means. The fastening means is defined to be the
`means of attaching the vibrotactile unit to a sensing body
`part, transmitting (and possibly modifying) the vibrations
`created by the vibrotactile unit. This means may beonethat
`is flexible such as a strap made of cloth or soft polymer, or
`rigid, such as metal or hard polymer which grabs or pinches
`the flesh, skin or hair. The fastening means mayalso include
`gluing or taping to the skin or hair, or tying with a string or
`rope around a limb, or attaching to clothes with Velcro® or
`similarly functional means. A vibrotactile unit may also be
`attached to another structure which is then attached to the
`
`body part with the same means just mentioned. The vibra-
`tions generated by the actuator may be transmitted to the
`sensing body part by the structure (rigid or non-rigid), or
`through a linkage transmission or a fluid transmission.
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`23
`
`23
`
`

`

`6,088,017
`
`5
`The eccentric mass need not be mounted directly onto a
`motor shaft. A mechanical transmission mayrotate the mass
`on a different shaft than the motor shaft. The mass-moving
`actuator rotates this shaft. Fluids such as air and liquids may
`also transmit the motion from a power source to the rotating
`eccentric mass. Changing magnetic fields may also be
`employed to induce vibration of a ferrous mass.
`As previously mentioned, state signals may rclate to a
`physicalor virtual state. Whenthe state represents a physical
`condition, the subject invention includes a state measure-
`ment sensor which producesa state signal. This state mea-
`surement sensor may measure somepropertyof the sensing
`body part. Recall that the body part associated with receiv-
`ing the vibrotactile stimulation is called the sensing body
`part, the body part associated with producing the activating
`signal is called the measured bodypart. The signal processor
`may receive signals from this sensor such as a tactile,
`position, bend, velocity, acceleration or temperature sensor
`and generate an activating signal. In this way, the user may
`receive feedback basedonhis actionsor physical state. For
`example, the vibrotactile device may be used to train the
`user to do some physical motion task. In this case,
`the
`position or motion of the body part whichis to do the motion
`task is measured by the state measurementsensorand is also
`the sensing body part. Direct stimulation to the body part
`being trained enhances the training of the task. Complex
`actuation in the form of a function of different levels of
`
`frequency or amplitude may inform the user whether his
`actions are correct or incorrect; the level of correctness may
`correspond to the level of frequency or amplitude.
`In addition,
`the sensing body part (which is also the
`measured body part) may have a graphical representation
`shown to the user. The uscr may also be presented with
`visual, auditory, taste, smell, force and/or temperature cues
`to his actions in combination with the vibrotactile cues
`
`invention. The user may be
`provided by the subject
`immersed in a virtual environment. The user may see a
`graphical representation of his/her bodypart interact with
`virtual objects and simultaneously feel a corresponding
`tactile sensation simulating the interaction. For example a
`user may havehis/her fingers be the sensing and measured
`body parts. The user may then see his/her virtual hand in the
`virtual environmentcontact a virtual object. The user would
`then feel an increasing vibratory stimulation on his/her
`physical fingertip as he increased the virtual pressure on the
`virtual object using the virtual fingertip.
`As previously discussed, using the vibrotactile device of
`the subject invention, a user may receive tactile sensations
`based on the state of his body parts. In the previous case the
`state included the position, and other dynamic quantities, of
`the body parts. In certain applications, the measured body
`part is the sameas the sensing bodypart(the list of possible
`sensing body parts mentioned earlier also applies to mea-
`sured body parts); in other applications they are different
`body parts. When the measured body part is different than
`the sensing body part,
`the subject
`invention acts as a
`coupling device which relates the sensing bodypart and the
`measured body part.
`In another application, the user may receivetactile feed-
`back as a result of the conditions of a computer simulated
`environment, not necessarily related to the user’s aclions or
`state. The vibrotactile units with varying actuation levels
`may be used to simulate a variety of contact situations, e.g.,
`contact with fluids and solids, and contacts which are
`momentary or continuous. For example, a user immersed in
`a computer simulated virtual environment may feel simu-
`lated fluid (like air or water) across his body. In such a
`
`10
`
`15
`
`,,
`
`,
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`simulation, an arcay of vibrotactile units may vibrate in
`sequence to correspond to a pressure wave hitting the
`corresponding parts of the body;
`the amplitude of the
`vibration may vary to correspond to different
`levels of
`pressure being simulated. A user may also feel a virtual
`object that comes into contact with a portion of his virtual
`body. The user mayfeel a virtual bug crawl up his virtual
`arm by sequencing anarray of vibrotactile units. To accom-
`pany the tactile sensations received by the user which are
`uncorrelated with his actions, the user may be presented with
`visual, auditory, taste, smell, force, temperature and other
`forms of feedback in order to enhance the realism of the
`simulated environment.
`
`In yet another application of the vibrotactile device, a
`group of users may receive tactile sensations.
`In one
`example, users may wear individual vibrotactile units, or
`they may also share vibrotactile units as follows. A tactile
`sensation may be shared by one or more users making
`physical contact with the sensing body part of another user.
`Tor example, one user may wear vibrotactile units on the
`backs of his fingers. A second user, not wearing any vibro-
`tactile units, may obtain vibrotactile feedback transmitted
`via the first user when the first user places the palmar side
`of his fingers on a sensing bodypart of the second user. The
`activating signal for each vibrotactile unit may be computer
`controlled via either user’s actions or through a computer
`simulated event. In a second example, a group of users may
`each receive identical tactile feedback through individually
`mounted vibrotactile units. The commonactivating signal
`may correspond to measured bodyparts from a single,
`optionally separate, user. Different users may also be respon-
`sible for producing the commonactivating signal for one or
`more vibrotactile units. For instance, the movement of one
`user’s arm may control the vibrotactile unit on each user’s
`arm; and the voice of a second user may control
`the
`vibrotactile unit on each user’s back; the eye-gaze of three
`other users may controlthe vibrotactile unit upon which they
`stare in unison. An example application of a single user
`controlling manyuser’s vibrotactile sensations is a new form
`of entertainment where a performer creates vibrotactile
`sensations for an audience.
`
`the vibrotactile units are
`In a preferred embodiment,
`affixed to an instrumented glove, such as the CyberGlove™
`manufactured by Virtual Technologies of Palo Alto, Calif.,
`USA. The CyberGlove has sensors in it which measure the
`angles of the joints of the hand.
`‘The fingertips of the
`CyberGlove are open so that the user may reliably handle
`physical objects while wearing the glove. The open finger-
`tips allowthe user to feel the sensations of real objects in
`conjunction with the generated vibrotactile sensations. The
`fingertips need not be open, they may be fully enclosed as
`in the 22-sensor model of the CyberGlove. The mass-
`moving actuator of each vibrotactile unit is encased in a
`cylindrical housing and mounted onto the glove on each of
`the fingers and thumb, and on the palmarside of the hand.
`Each mass-moving actuator is composed of a small DC
`motor with an eccentric mass mounted rigidly onto the shaft
`of the motor. The casing is made of tubular plastic and serves
`to protect the motion of the mass from the user and protect
`the user from the rotating mass. The casing may be made of
`anyrigid or semi-rigid material including but not limited to
`steel, aluminum, brass, copper, plastic,
`rubber, wood,
`composite, fiberglass, glass, cardboard, and the like. The
`casing may form a solid barrier, a wire-mesh, grid or
`column-like support capable of transmitting vibrations from
`the mass-moving actuator to the fastening means. The
`instrumented glove informs a computerofthe position ofthe
`
`24
`
`24
`
`

`

`6,088,017
`
`7
`user’s hand and fingers. The computer, whichis part of the
`signal processor, then interprets this hand state signal (and
`any virtual state signal if the application calls for it). The
`computer then generates a control signal, which when pro-
`cessed by the driver, activates the actuators to create tactile
`sensations.
`
`One feature of the embodiment of the subject invention
`just described, which employs an cccentric mass,is that the
`energy imparted into the system can be less than the energy
`required when using electromagnetic coils (such as the
`speaker voice coils used by Patrick et al. and the EXOS
`TouchMaster). Encrgy is stored as rotational incrtia in the
`eccentric mass, whereas the voice-coil-based systems lose
`all inertial energy each time the coil change directions.
`Another feature of the subject inventionis that vibrating
`the bone structure of a body part, as well as skin
`mechanoreceptors, has an advantage over stimulating just
`the skin mechanoreceptors (such as Meissner, Merkel,
`Ruffini and Pacinian corpuscles) in that the nerves do notget
`easily overstimulated and do not become numb. In addition,
`the form of information to the user is closer to a physical
`contact sensation where the muscles and joints are
`stimulated, as is done by full force feedback systems. As a
`result, the vibrotactile units need not be attached to a body
`part which has sensitive skin mechanoreceptors. For
`example, a vibrotactile unit may be attached to a fingernail
`or an elbow.
`In an embodiment in which a user is immersed in a
`
`computer simulated environment, actuation of vibrotactile
`units can approximate the sensation of touching physical
`objects as full force feedback devices do. The deep impul-
`sive sensation in the muscles and joints generated bythe
`vibrotactile units simulates the change in proprioceptive
`state as the user touches a virtual object. The subject
`invention provides numerous advantages over a sustained
`force feedback device. For example, because of its
`simplicity, the vibrotactile device of the subject invention
`can be made smaller,
`lighter,
`less encumbering, more
`robustly and cheaper.
`The subject invention may be used in combination with a
`sustained force feedback device as provided by Kramer in
`US. Pat. No. 5,184,319, Kramer in U.S. patent application
`Ser. No. 08/373,531 (allowed), Zarudiansky in U-S. Pat. No.
`4,302,138, Burdea in U.S. Pat. No. 5,354,162, and Jacobus
`in US. Pat. No. 5,389,865. These patents and patent appli-
`cations are incorporated herein by reference. Such a com-
`bination can give a higher frequency response than that
`capable of being generated by the sustained force feedback
`device and/or to reduce the cost and/or size of the full
`system. The subject invention may also be used in combi-
`nation with other tactile feedback devices such as heating or
`cooling, devices, bladder devices or voice coils.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1a is a perspective view of an electric mass-moving
`actuator with an eccentric mass attached to its shaft.
`
`FIG. 1b is a perspective view of a mass-moving linear
`actuator with a mass attached to its shaft.
`FIGS. 2a and 2b are a cross-sectional side view and a
`perspective view respectively of an example of a vibrotactile
`unit.
`
`FIG. 3 is a perspective view ofthe vibrotactile unit shown
`in FIG. 2b where the vibrotactile unit is attached to the
`
`palmarside of the fingertip.
`FIG. 4 is another perspective view of a vibrotactile unit
`attached to the dorsal side of the fingertip, where it makes
`contact with the nail.
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`FIG. 5 is a perspective view ofa vibrotactile unit attached
`to the dorsal side of the proximal phalanx.
`FIG. 6 is a perspective view of a vibrotactile unit attached
`to the palm of the hand
`FIG. 7 is a perspective view ofa vibrotactile unit attached
`to the dorsal side of the metacarpus (the back of the hand).
`FIG. 8 is a perspective view ofa vibrotactile unit attached
`to the top of the foot.
`FIG. 9 is a side view of a multitude of vibrotactile units
`
`attached to a variety of places on the head.
`FIGS. 10a@ and 105 are front and back views respectively,
`of a multitude of vibrotactile units attached to a variety of
`places on the body.
`FIGS. lla and 115 are perspective and front views
`respectively of a fastening means where the palmar side of
`the fingertip receives greater stimulation without the vibro-
`tactile u