(12) United States Patent
`Martin et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,563,487 B2
`May 13, 2003
`
`US006563487B2
`
`(54)
`
`(75)
`
`HAPTIC FEEDBACK FOR DIRECTIONAL
`CONTROL PADS
`
`Inventors: Kenneth M. Martin, Palo Alto, CA
`(US); Louis B. Rosenberg, San Jose,
`CA (US)
`
`(73)
`
`Assignee: Immersion Corporation, San Jose, CA
`(Us)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21)
`
`(22)
`(65)
`
`(63)
`
`(51)
`
`(52)
`
`(58)
`
`(56)
`
`Appl. No.: 09/467,309
`
`Filed:
`
`Dec. 17, 1999
`
`Prior Publication Data
`
`US 2003/0030619 A1 Feb. 13, 2003
`
`Related US. Application Data
`
`Continuation-in-part of application No. 09/253,132, ?led on
`Feb. 18, 1999, now Pat. No. 6,243,078, which is a continu
`ation-in-part of application No. 09/156,802, ?led on Sep. 17,
`1998, now Pat. No. 6,184,868, which is a continuation-in
`part of application No. 09/103,281, ?led on Jun. 23, 1998,
`now Pat. No. 6,088,019.
`
`Int. Cl.7 ................................................ .. G09G 5/00
`
`US. Cl. ..................................................... .. 345/156
`
`Field of Search ............................... .. 345/156, 161,
`345/168, 169; 463/37, 38
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,896,125 A * 4/1999 NiedZwiecki ............. .. 345/168
`5,903,257 A * 5/1999 Nishiumi et al. .
`..
`
`
`
`5,984,785 A * 11/1999 Takeda et al. 6,102,803 A * 8/2000 Takeda et al. .
`
`OTHER PUBLICATIONS
`
`Baigrie, “Electric Control Loading —A Low Cost, High
`Performance Alternative,” Proceedings, pp. 247—254, Nov.
`6—8, 1990.
`Environment,”
`Virtual
`Iwata,
`“Pen—based Haptic
`0—7803—1363—1/93 IEEE, pp. 287—292, 1993.
`Russo, “The Design and Implementation of a Three Degree
`of Freedom Force Output Joystick,” MIT Libraries Archives
`Aug. 14, 1990, pp. 1—131, May 1990.
`Brooks et al., “Hand Controllers for Teleoperation —A
`State—of—the—Art Technology Survey and Evaluation,” JPL
`Publication 85—11; NASA—CR—175890; N85—28559, pp.
`1—84, Mar. 1, 1985.
`Jones et al., “A perceptual analysis of stiffness,” ISSN
`0014—4819 Springer International (Springer—Verlag);
`Experimental Brain Research, vol. 79, No. 1, pp. 150—156,
`1990.
`
`(List continued on next page.)
`
`Primary Examiner—Richard Hjerpe
`Assistant Examiner—Ronald Laneau
`(74) Attorney, Agent, or Firm—Kilpatrick Stockton LLP
`(57)
`ABSTRACT
`
`Ahaptic feedback control device coupled to a host computer
`and outputting forces to a user of the control device. The
`control device includes a housing, a direction pad capable of
`being contacted by the user in at least two different locations
`to provide two different directional signals to the host
`computer, and a computer-controlled actuator that outputs a
`force directly on the direction pad. A sensor, such as one or
`more contact switches, can detects when the locations of the
`direction pad have been contacted or pressed by the user.
`The actuator can be a linear or rotary actuator that outputs
`a force on the direction pad, e.g. approximately perpendicu
`lar to the top surface of the direction pad. The actuator can
`output a vibration or a pulse tactile sensation on the direction
`pad in coordination with interactions or events in a computer
`graphical environment or functions of a controlled electronic
`device. The control device can be a game controller, a
`mouse, a remote control device, or other type of device.
`
`6,225,976 B1 * 5/2001 Yates et al. ............... .. 345/156
`
`60 Claims, 6 Drawing Sheets
`
`SCEA Ex. 1009 Page 1
`
`

`

`US 6,563,487 B2
`Page 2
`
`OTHER PUBLICATIONS
`
`Burdea et al., “Distributed Virtual Force Feedback, Lecture
`Notes for Workshop on Force Display in Virtual Environ
`ments and its Application to Robotic Teleoperation,” 1993
`IEEE International Conference on Robotics and Automa
`tion, pp. 25—44, May 2, 1993.
`SnoW et al., “Model—X Force—Re?ecting—Hand—Control
`ler,” NT Control No. MPO—17851; JPL Case No. 5348, pp.
`1—4, Jun. 15, 1989.
`Ouh—Young, “Force Display in Molecular Docking,” Order
`No. 9034744, pp. 1—369, 1990.
`Tadros, Control System Design for a Three Degrees of
`Freedom Virtual Environment Simulator Using Motor/
`Brake Pair Actuators, MIT Archive @ Massachusetts Insti
`tute of Technology, pp. 1—88, Feb. 1990.
`CaldWell et al., “Enhanced Tactile Feedback (Tele—Taction)
`Using a Multi—Functional Sensory System,” 1050—4729/93,
`pp. 955—960, 1993.
`Adelstein, “Design and Implementation of a Force Re?ect
`ing Manipulandum for Manual Control research,” DSC—vo1.
`42, Advances in Robotics, Edited by H. KaZerooni, pp.
`1—12, 1992.
`GotoW et al., “Controlled Impedance Test Apparatus for
`Studying Human Interpretation of Kinesthetic Feedback,”
`WA11—11:00, pp. 332—337.
`Stanley et al., “Computer Simulation of Interacting Dynamic
`Mechanical Systems Using Distributed Memory Parallel
`Processors,” DSC—vo1. 42, Advances in Robotics, pp.
`55—61, ASME 1992.
`Russo, “Controlling Dissipative Magnetic Particle Brakes in
`Force Re?ective Devices,” DSC—vo1. 42, Advances in
`Robotics, pp. 63—70, ASME 1992.
`Kontarinis et al., “Display of High—Frequency Tactile Infor
`mation to Teleoperators,” Telemanipulator Technology and
`Space Telerobotics, Won S. Kim, Editor, Proc. SPIE vol.
`2057, pp. 40—50, Sep. 7—9, 1993.
`Patrick et al., “Design and Testing of A Non—reactive,
`Fingertip, Tactile Display for Interaction With Remote Envi
`ronments,” Cooperative Intelligent Robotics in Space, Rui J.
`deFigueiredo et al., Editor, Proc. SPIE vol. 1387, pp.
`215—222, 1990.
`Adelstein, “A Virtual Environment System for the Study of
`Human Arm Tremor,” Ph.D. Dissertation, Dept. of Mechani
`cal Engineering, MIT, Jun. 1989.
`BejcZy, “Sensors, Controls, and Man—Machine Interface for
`Advanced Teleoperation,” Science, vol. 208, No. 4450, pp.
`1327—1335, 1980.
`BejcZy, “Generalization of Bilateral Force—Re?ecting Con
`trol of Manipulators,” Proceedings of Fourth CISM—IF
`ToMM, Sep. 8—12, 1981.
`McAffee, “Teleoperator Subsystem/Telerobot Demonsdtra
`tor: Force Re?ecting Hand Controller Equipment Manual,”
`JPL D—5172, pp. 1—50, A1—A36, B1—B5, C1—C36, Jan.
`1988.
`Minsky, “Computational Haptics: The Sandpaper System
`for SynthesiZing TeXture for a Force—Feedback Display,”
`Ph.D. Dissertation, MIT, Jun. 1995.
`Jacobsen et al., “High Performance, DeXtrous Telerobotic
`Manipulator With Force Re?ection,” Intervention/ROV ’91
`Conference & Exposition, HollyWood, Florida, May 21—23,
`1991.
`
`Shimoga, “Finger Force and Touch Feedback Issues in
`DeXterous Telemanipulation,” Proceedings of Fourth
`Annual Conference on Intelligent Robotic Systems for
`Space EXpploration, Rensselaer Polytechnic Institute, Sep.
`30—Oct. 1, 1992.
`IBM Technical Disclosure Bullein, “Mouse Ball—Actuating
`Device With Force and Tactile Feedback,” vol. 32, No. 9B,
`Feb. 1990.
`Terry et al., “Tactile Feedback in a Computer Mouse,
`Proceedings of Fouteenth Annual Northeast Bioengineering
`Conference, University of NeW Hampshire, Mar. 10—11,
`1988.
`HoWe, “A Force—Re?ecting Teleoperated Hand System for
`the Study of Tactile Sensing in Precision Manipulation,”
`Proceedings of the 1992 IEEE International Conference on
`Robotics and Automation, Nice, France, May 1992.
`Eberhardt et al., “Omar —A Haptic display for speech
`perception by deaf and deaf—blind individuals,”IEEE Virtual
`Reality Annual International Symposium, Seattle, WA, Sep.
`18—22, 1993.
`RabinoWitZ et al., “Multidimensional tactile displays: Iden
`ti?cation of vibratory intensity, frequency, and contactor
`area,” Journal of the Acoustical Society of America, vol. 82,
`No. 4, Oct. 1987.
`BejcZy et al., “Kinesthetic Coupling BetWeen Operator and
`Remote Manipulator,” International Computer Technology
`Conference, The American Society of Mechanical Engi
`neers, San Francisco, CA, Aug. 12—15, 1980.
`BejcZy et al., “A Laboratory Breadboard System for
`Dual—Arm Teleoperation,”SOAR ’89 Workshop, JSC,
`Houston, TX, Jul. 25—27, 1989.
`Ouh—Young, “A LoW—Cost Force Feedback Joystick and Its
`Use in PC Video Games,” IEEE Transactions on Consumer
`Electronics, vol. 41, No. 3, Aug. 1995.
`Marcus, “Touch Feedback in Surgery,” Proceedings of Vir
`tual Reality and Medicine The Cutting Edge, Sep. 8—11,
`1994.
`BejcZy, et al., “Universal Computer Control System (UCCS)
`for Space Telerobots,” CH2413—3/87/0000/0318501.00
`1987 IEEE, 1987.
`Patrick, “Design, Construction, and Testing of a Fingetip
`Tactile Display for Interaction With Virtual and Remote
`Environments,” Master of Science Thesis, MIT, Nov. 8,
`1990.
`Cadler, “Design of A Force—Feedback Touch—Introducing
`Actuator for Teleoperator Robot Control,” Bachelor of Sci
`ence Thesis, MIT, Jun. 23, 1983.
`Wiker, “Teletouch Display Development: Phase 1 Report,”
`Technical Report 1230, Naval Ocean Systems Center, San
`Diego, Apr. 17, 1989.
`Bliss, “Optical—to—Tactile Image Conversion for the Blind,”
`IEEE Transactions on Man—Machine Systems, vol.
`MMS—11, No. 1, Mar. 1970.
`Johnson, “Shape—Memory Alloy Tactile Feedback Actua
`tor,” Armstrong Aerospace Medical Research Laboratory,
`AAMRL—TR—90—039, Aug., 1990.
`
`* cited by examiner
`
`SCEA Ex. 1009 Page 2
`
`

`

`U.S. Patent
`
`May 13, 2003
`
`Sheet 1 6f 6
`
`US 6,563,487 B2
`
`SCEA Ex. 1009 Page 3
`
`

`

`U.S. Patent
`
`May 13, 2003
`
`Sheet 2 6f 6
`
`US 6,563,487 B2
`
`18 @@
`
`SCEA Ex. 1009 Page 4
`
`

`

`U.S. Patent
`
`May 13, 2003
`
`Sheet 3 6f 6
`
`US 6,563,487 B2
`
`FIG. 2b
`
`SCEA Ex. 1009 Page 5
`
`

`

`U.S. Patent
`
`May 13, 2003
`
`Sheet 4 6f 6
`
`US 6,563,487 B2
`
`SCEA Ex. 1009 Page 6
`
`

`

`U.S. Patent
`
`May 13, 2003
`
`Sheet 5 6f 6
`
`US 6,563,487 B2
`
`FIG. 5
`
`146
`
`FIG. 6
`
`"6
`
`SCEA Ex. 1009 Page 7
`
`

`

`U.S. Patent
`
`May 13, 2003
`
`Sheet 6 6f 6
`
`US 6,563,487 B2
`
`198
`
`198
`
`196
`
`SCEA Ex. 1009 Page 8
`
`

`

`US 6,563,487 B2
`
`1
`HAPTIC FEEDBACK FOR DIRECTIONAL
`CONTROL PADS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation-in-part of parent patent
`applications:
`Application Ser. No. 09/156,802, now US. Pat. No.
`6,184,868, ?led Sep. 17, 1998 entitled, “Haptic Feed
`back Control Devices,”
`Application Ser. No. 09/103,281, now US. Pat. No.
`6,088,019, ?led Jun. 23, 1998 entitled, “LoW Cost
`Force Feedback Device With Actuator for Non-Primary
`Axis,” and
`Application Ser. No. 09/253,132, now US. Pat. No.
`6,243,078, ?led Feb. 18, 1999 on behalf of Louis
`Rosenberg, entitled, “Pointing Device With Forced
`Feedback Button,”
`all assigned to the assignee of this present application, and
`all of Which are incorporated herein by reference in their
`entirety.
`
`BACKGROUND OF THE INVENTION
`
`The present invention relates generally to the interfacing
`With computer and mechanical devices by a user, and more
`particularly to devices used to interface With computer
`systems and electronic devices and Which provide haptic
`feedback to the user.
`Humans interface With electronic and mechanical devices
`in a variety of applications, and the need for a more natural,
`easy-to-use, and informative interface is a constant concern.
`In the context of the present invention, humans interface
`With computer devices for a variety of applications. One
`such application is interacting With computer-generated
`environments such as games, simulations, and application
`programs. Computer devices such as game consoles (e.g.
`available from Nintendo, Sony, and Sega), personal
`computers, portable computers, and internet appliances are
`typically used to provided these environments, although a
`Wide variety of devices can be used.
`For many computer generated environments, users inter
`face With the environment using an interface device having
`controls such as a joystick, gamepad, mouse, trackball,
`buttons, steering Wheel, or other device physically contacted
`and manipulated by the user. The user manipulates the
`controls Which provides position or other related input data
`to the computer, and the computer updates the environment
`or program in response, such as a game program.
`In some interface devices, force feedback or tactile feed
`back is also provided to the user, more generally knoWn
`herein as “haptic feedback.” In the game console market,
`haptic feedback interface devices such as “gamepad” con
`trollers (or add-on hardWare for gamepad controllers) for
`game consoles include the Dual Shock from Sony, the
`Rumble Pack from Nintendo, and the Jump Pack from Sega.
`These devices are vibrotactile-feedback controllers Which
`employ one or more motors coupled to the housing of the
`controller to shake the housing and thus provide output
`vibrations to the user Which are correlated to game events
`and interactions. Typically, an eccentric rotating mass
`(ERM) motor is used to generate vibration on the controller
`housing and thus to the user.
`One problem With existing vibrotactile controllers is that
`the vibrations produced are generaliZed in location to the
`entire housing of the control device. Thus the user cannot
`
`2
`experience localiZed tactile sensations that are speci?c to a
`portion of the gamepad controller or to a control on the
`controller, such as a button or direction pad (“D-pad”). Thus,
`only one type of tactile sensation can be experienced at one
`time by the user, Which limits the immersive experience for
`the user When interacting With the computer.
`
`SUMMARY OF THE INVENTION
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`The present invention is directed to a loW-cost haptic
`feedback control on an interface device connected to a
`computer system, Where the control is a directional pad
`provided With haptic sensations for enhancing interactions
`and manipulations in a displayed graphical environment or
`When controlling an electronic device.
`More speci?cally, the present invention relates to a haptic
`feedback control device coupled to a host computer and
`outputting forces to a user of the control device, the control
`device including a housing that can in some embodiments be
`held and operated by a user in one or more hands. Adirection
`pad is coupled to the housing and is capable of being
`contacted by the user in at least tWo different locations to
`provide tWo different directional signals to the host
`computer, each directional signal corresponding to one of
`the locations on the direction pad. A computer-controlled
`actuator is coupled to the direction pad and outputs a force
`directly on the pad.
`In one embodiment, each of the locations on the direction
`pad is an extension of the direction pad, the extensions being
`part of a unitary member, Wherein the unitary member tilts
`approximately about a pivot point When the user presses one
`of the locations. In other embodiments, each of the locations
`on the direction pad is a button separate from a central
`portion of the direction pad. A sensor, such as one or more
`contact sWitches, detects When the locations of the direction
`pad have been contacted or pressed by the user. An elasto
`meric layer positioned under the direction pad can provide
`conductive portions to engage the contact sWitches. The
`direction pad is preferably capable of being contacted by the
`user in four different locations, each location providing a
`different directional signal to the host computer.
`The actuator can be a linear actuator that provides an
`output force in a linear degree of freedom, e.g. approxi
`mately perpendicular to the top surface of the direction pad,
`or a rotary actuator that provides an output force in a rotary
`degree of freedom, Where at least one member couples the
`actuator to the directional pad Which converts the rotary
`output force to an approximately linear output force
`imparted on the direction pad. The actuator can be a voice
`coil actuator, a pieZo-electric actuator, a pager motor, a
`solenoid, or other type. The actuator can output a vibration
`or a pulse tactile sensation on the direction pad. The com
`puter displays a graphical environment Which With the user
`interacts using the control device, and the control device can
`be a game controller, a mouse, a remote control device, or
`other type of device.
`A microprocessor separate from the computer can receive
`force information from the host computer and provide
`control signals based on the force information to control the
`actuator. The user can interact With a graphical environment
`using said control device, such as a game, Web page, or
`graphical user interface. In some embodiments, a sensor can
`be used to detect motion or position of the direction pad
`approximately perpendicularly to a top surface of the direc
`tion pad, Wherein an input signal based on the detected
`motion or position is sent to the computer. In some
`embodiments, the direction pad is caused to move to a loWer
`
`SCEA Ex. 1009 Page 9
`
`

`

`US 6,563,487 B2
`
`3
`position When a user-controlled graphical object is moved
`over a selectable graphical object, to make the selection
`easier. Or, the direction pad can be moved to different
`positions on an axis approximately perpendicular to a top
`surface of the direction pad, the different positions based on
`a desired 3-D elevation of an object or area displayed in the
`graphical environment. A method for providing haptic feed
`back to a direction pad of an interface device includes
`similar features.
`The present invention advantageously provides haptic
`feedback device to a direction pad of an interface device.
`The direction pad haptic feedback can be independent of any
`other haptic feedback provided for the device, alloWing a
`greater variety of tactile sensations and different tactile
`sensations to be output simultaneously. The haptic feedback
`direction pad is signi?cantly loWer in cost than many other
`types of haptic feedback devices and is thus quite suitable
`for home consumer applications. The embodiments
`described alloW crisp forces and a variety of sensations to be
`output to enhance the user’s interaction and experience With
`a computer application or electronic device.
`These and other advantages of the present invention Will
`become apparent to those skilled in the art upon a reading of
`the folloWing speci?cation of the invention and a study of
`the several ?gures of the draWing.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1a is a perspective vieW of a gamepad control
`including the directional pad of the present invention;
`FIG. 1b is a perspective vieW of a remote control device
`including the directional pad of the present invention;
`FIGS. 2a and 2b are perspective exploded vieW and side
`elevational vieWs, respectively, of a ?rst embodiment of a
`direction pad of the present invention for providing haptic
`feedback to the user;
`FIG. 3a is a perspective vieW of a different embodiment
`of the direction pad of the present invention, Which is
`integrated With an elastomeric layer and a printed circuit
`board in a sub-assembly;
`FIGS. 3b, 3c, 3d, and 3e illustrate different actuators for
`use With the direction pad of the present invention;
`FIG. 4 is a side elevational vieW of a different embodi
`ment of the direction pad of the present invention in Which
`a linear actuator is used to provide linear motion along an
`axis different from the Z-axis;
`FIG. 5 is a side elevational vieW of a different embodi
`ment of the direction pad of the present invention in Which
`the direction pad provides analog input based on the amount
`it is moved;
`FIG. 6 is a perspective vieW of a different embodiment of
`the direction pad of the present invention in Which a coil
`routed around the direction pad and multiple magnets pro
`vide haptic feedback on the direction pad; and
`FIG. 7 is a side elevational vieW of a different embodi
`ment of the present invention in Which a pieZo-electric
`actuator is directly coupled to the direction pad of the
`present invention.
`
`15
`
`25
`
`35
`
`45
`
`55
`
`DETAILED DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`FIG. 1a is a perspective vieW of one example of a force
`feedback interface device 10 including the active tactile
`direction pad of the present invention. The device is used for
`interfacing a user With a computer generated environment
`implemented by a host computer 12.
`
`65
`
`4
`Interface device 10 of the described embodiment is in the
`form of a handheld controller, of similar shape and siZe to
`many “gamepads” currently available for video game con
`sole systems. A housing 14 of the interface device 10 is
`shaped to easily accommodate tWo hands gripping the
`device at the gripping projections 16a and 16b. In the
`described embodiment, the user accesses the various con
`trols on the device 10 With his or her ?ngers. In alternate
`embodiments, the interface device can take a Wide variety of
`forms, including devices that rest on a tabletop or other
`surface, stand-up arcade game machines, laptop devices or
`other devices Worn on the person, handheld or used With a
`single hand of the user, etc.
`A direction pad (“d-pad” or “joypad”) 18 is included on
`device 10 to alloW the user to provide directional input to the
`host computer 12. In its most common implementation, the
`direction pad 18 is approximately shaped like a cross or disc
`having four extensions or directional positions (all referred
`to as “extensions” herein) radiating from a central point at a
`90-degree spacing, Where the user can press doWn on one of
`the extensions 20 to provide a directional input signal to the
`host computer for the corresponding direction. Typically, the
`direction pad 18 is tilted approximately about a pivot point,
`eg when one of the extension is pressed, the entire pad tilts
`as a single unit. In other embodiments, just the extension
`portion 20 that is pressed can move relative to the other parts
`of pad 18. In yet other embodiments, a contact sensor can be
`provided in the pad 18 at each extension 20 so that the
`presence of a ?nger is detected With mechanical sWitches
`and no tilting motion or pressing motion is required.
`TWo adjacent extensions 20 can be simultaneously
`pressed to provide input for the appropriate diagonal direc
`tion betWeen the extensions. In other embodiments, the
`direction pad 18 can include eight extensions or other
`number of extensions. The directional input is most com
`monly used to move a user-controlled graphical object in the
`direction corresponding to the direction of the extension,
`e.g. pressing on the “up” extension moves a cursor or
`character up or forWard, pressing “left” moves the cursor or
`character left, etc. Also, in some embodiments, the direction
`pad 18 can be pressed doWn as a single piece (e.g., from the
`center) to provide a different input signal than the signals
`provided When pressing the extensions 20. For example, the
`entire pad 18 can be pressed to provide the equivalent of a
`single mouse-click to select an object pointed to by a cursor.
`A contact sWitch can be positioned at the end of the
`doWnWard travel of the pad to provide the signal, or other
`types of sWitches or sensors can be used (optical, magnetic,
`etc.).
`In the present invention, the direction pad 18 is provided
`With the ability to output haptic feedback such as tactile
`sensations to the user Who is contacting the direction pad 18.
`Various embodiments detailing the structure of the haptic
`feedback direction pad are described in greater detail beloW.
`Preferably, the forces output on the pad are linear and along
`the Z-axis, approximately perpendicular to the top surface of
`the pad 18. Herein, the “top surface” of the pad is generally
`meant to refer to the overall x-y orientation of the surface of
`the pad, since there may be several actual top surfaces, eg
`on a pad having bumps or ridges on its top surface. Using
`one or more actuators and sensors coupled to the direction
`pad (or other controls), a variety of force sensations can be
`output to the user Who is contacting the button. For example,
`jolts, vibrations (varying or constant amplitude), and tex
`tures can be output. Forces can be at least in part based on
`the position of the direction pad in a degree of freedom,
`and/or on the location or state of a controlled object in the
`
`SCEA Ex. 1009 Page 10
`
`

`

`US 6,563,487 B2
`
`5
`graphical environment of the host computer, and/or inde
`pendent of direction pad position or obj ect state. Such forces
`output on the pad 18 are considered “computer-controlled”
`since a microprocessor or other electronic controller is
`controlling the magnitude and/or direction of the force
`output of the actuator(s) using electronic signals. In some
`embodiments, the entire pad 18 is provided With haptic
`sensations as a single unitary member; in other
`embodiments, some or all of extensions 20 can each be
`provided With its oWn haptic feedback actuator and related
`transmissions so that haptic sensations can be provided for
`only a particular extension 20 (and in some embodiments the
`center portion of the pad 18 can include its oWn actuator for
`sensations independent of the sensations provided for each
`extension 20).
`One advantage of the present invention is that haptic
`sensations can be output on the direction pad even When the
`direction pad is being held at a limit to its movement by the
`user. In joystick devices, the device is often unable to output
`a haptic sensation to the joystick handle When the handle is
`being held at a limit by the user, eg when the user has
`moved the joystick fully in one direction during a game. This
`is because the physical stop Which the joystick is held
`against prevents motion of the stick in that direction.
`HoWever, in the present direction pad invention, the direc
`tion pad can be held continuously in one direction (eg one
`or more extensions 20 can be continuously pressed) and
`haptic sensations can still be output, since the preferred
`haptic direction is along the Z-axis. Thus there is no loss of
`haptic ?delity When using the direction pad for input to the
`host computer.
`In some embodiments, the direction pad can also or
`alternatively provide a proportional input to a host computer,
`indicating the distance or amount that an extension of the
`pad is pushed. Forces can be output on the control by an
`actuator in the device 10 as the user moves it. These forces
`can be colocated such that the user feels the forces in the
`degree of freedom of movement of the pad extension. For
`example, a spring or damping resistance force can be applied
`to resist movement of the extension (or to resist motion of
`the entire pad 18 if pressing it from the center of the pad 18).
`Also, in a different alternate embodiment, forces can be
`applied to the direction pad 18 to cause side-to-side (e.g.,
`x-y) motion of the pad in addition to or instead of Z-axis
`motion. For example, one linear actuator can provide motion
`for the x-axis, and a second linear actuator can provide
`motion for the y-axis.
`One or more buttons 24 can also be provided on the
`surface of the housing 14 of the device 10. The user’s hands
`have easy access to the buttons, each of Which may be
`pressed by the user to provide a distinct input signal to the
`host computer 12. In some embodiments, one or more of the
`buttons 24 can be provided With tactile feedback similarly to
`the embodiments of the direction pad 18 described beloW.
`A ?nger joystick 26 can be included in device 10 that
`projects out of the top surface of the housing 14 to be
`manipulated by the user in one or more degrees of freedom.
`The user can grasp each of grips 16a and 16b of the device
`and use a thumb or ?nger to manipulate the joystick 26 in
`tWo degrees of freedom (or three or more degrees of freedom
`in some embodiments). This motion is translated into input
`signals provided to the host computer 12, and can be
`different signals than those provided by the direction pad 18.
`In some embodiments, a additional linear or spin degrees of
`freedom can be provided for the joystick. In other
`embodiments, a sphere can be provided instead of or in
`addition to the joystick 26, Where one or more portions of the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`sphere can extend out of left, right, top and/or bottom sides
`of the housing 14, e.g., alloWing at least tWo ?ngers of the
`user to grip the sphere. The sphere may be rotated in place
`Within tWo rotary degrees of freedom and operate similarly
`to a joystick, as described in detail in copending provisional
`application No. 60/133,208, incorporated herein by refer
`ence.
`Instead of or in addition to buttons 26 and direction pad
`28, other controls may be placed Within easy reach of the
`hands grasping the housing 14. For example, one or more
`trigger buttons can be positioned on the underside of the
`housing and can be pressed by the ?ngers of the user. Other
`controls can also be provided on various locations of the
`device 10, such as a dial or slider for throttle control in a
`game, a four- or eight-Way hat sWitch, knobs, trackballs, a
`roller or sphere, etc. Any of these controls can also be
`provided With haptic feedback, such as tactile feedback. For
`example, embodiments of buttons, direction pads, and knobs
`having force feedback are described in copending patent
`applications Ser. No. 09/156,802, ?led Sep. 17, 1998, and
`Ser. No 09/179,382, ?led Oct. 26, 1998, all incorporated
`herein by reference. The forces can be colocated such that
`the user feels the forces in the degree of freedom of
`movement of the button or direction pad; or, the button,
`direction pad, or other control can be provided With tactile
`sensations such as vibrations.
`Furthermore, the housing itself and/or a moveable portion
`of the housing Which is contacted by the user When the user
`operates the device can provide tactile feedback as described
`in copending patent application Ser. No. 09/ 156,802, incor
`porated herein by reference. Thus, both the housing can
`provide tactile feedback (e. g., through the use of an eccentric
`rotating mass on a motor coupled to the housing) and the
`directional pad 18 can provide separate tactile feedback.
`This alloWs the host to control tWo different tactile sensa
`tions simultaneously to the user; for example, a vibration of
`a loW frequency can be conveyed through the housing to the
`user and a higher frequency vibration can be conveyed to the
`user through the directional pad 18. Each other button or
`other control provided With haptic feedback can also provide
`tactile feedback independently from the other controls.
`Interface device 10 is coupled to host computer 12 by a
`bus 32, Which can be any of several types of communication
`media. For example, a serial interface bus, parallel interface
`bus, or Wireless communication link can be used (radio,
`infrared, etc.). Speci?c implementations can include Uni
`versal Serial Bus (USB), IEEE 1394 (FireWire), RS-232, or
`other standards.
`The interface device 10 includes circuitry necessary to
`report control signals to the host computer 12 and to process
`command signals from the host computer 12. For example,
`sensors (and related circuitry) such as contact sWitches in
`elastomeric layers (described beloW) can be used to report
`the direction pressed by the user for the direction pad 18.
`Contact sWitches can be used as sensors for buttons, and
`potentiometers, optical encoders, or sWitches can be used to
`detect the position of the joystick and report that position to
`the host. The device also preferable includes circuitry that
`receives command signals from the host and outputs tactile
`sensations in accordance With the command signals using
`one or more device actuators. In some embodiments, a
`separate, local microprocessor can be provided on the device
`to both report sensor data to the host and to carry out
`commands received from the host, such commands
`including, for example, the type of tactile sensation and
`parameters describing the commanded tactile sensation. The
`microprocessor can implement tactile sensations indepen
`
`SCEA Ex. 1009 Page 11
`
`

`

`US 6,563,487 B2
`
`7
`dently after receiving a host command by controlling the
`device actuators; or, the host can maintain a greater degree
`of control over the tactile sensations by controlling actuators
`more directly. In other embodiments, logic circuitry such as
`state machines provided on the device 10 can handle tactile
`sensations as directed by the host computer 12. Architec
`tures and control methods that can be used for reading sensor
`signals and providing haptic feedback in the device 10 are
`described in greater detail in US. Pat. No. 5,734,373 and
`copending application Ser. Nos. 60/156,354, 60,133,208,
`09/376,649, and 60/160,401, all incorporated herein by
`reference.
`Host computer 12 is preferably a video game console,
`personal computer, Workstation, or other computing or elec
`tronic d