(12) United States Patent
`Fish
`
`I 1111111111111111 11111 lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`US006337678Bl
`US 6,337,678 Bl
`Jan.8,2002
`
`(10) Patent No.:
`(45) Date of Patent:
`
`(54) FORCE FEEDBACK COMPUTER INPUT
`AND OUTPUT DEVICE WITH
`COORDINATED HAPTIC ELEMENTS
`
`(75)
`
`Inventor: Daniel E. Fish, San Francisco, CA
`(US)
`
`(73) Assignee: Tactiva Incorporated, San Francisco,
`CA(US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`(21) Appl. No.: 09/357,727
`Jul. 21, 1999
`(22) Filed:
`
`Int. CI.7 .................................................. G09G 5/00
`(51)
`(52) U.S. CI .
`........................................ 345/156; 345/173
`(58) Field of Search ................................. 345/161, 156,
`345/157, 158, 173-179, 163
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5/1980 Kaplow et al. . ............ 364/900
`4,202,041 A
`10/1981 Pepper, Jr. ................... 178/18
`4,293,734 A
`4,385,366 A
`5/1983 Housey, Jr. . ................ 364/900
`5/1984 Ikeda et al. ... .. ... ... ... . .. 428/215
`4,448,837 A
`4,529,959 A
`7 /1985 Ito et al. . . .. ... ... .. ... ... . .. 338/295
`4/1990 Dunthom .................... 364/900
`4,914,624 A
`4/1990 Noda et al. ................... 382/59
`4,916,740 A
`10/1992 Asher .......................... 178/18
`5,159,159 A
`(List continued on next page.)
`OTIIER PUBLICKITONS
`Salisbury et al. "Haptic rendering programing Touch inter(cid:173)
`action with virtual objects," Symposium on Interactive 3D
`Techniques, Monterey, CA, Apr. 1995.*
`Affidavit of Daniel E. Fish Under 37 C.F.R. § 1.56 (dated
`Nov. 29, 1999).
`Bier, Eric A., Stone, Maureen C., Fishkin, Ken, Buxton,
`William, Baudel, Thomas, "A Taxonomy of See-Through
`Tools" (1994) (pp. 517-523). CHI94-4/94 Boston, MA.
`(List continued on next page.)
`
`Primary Examiner--Almis R. Jankus
`Assistant Examiner-Amr Awad
`or Firm--Skjerven
`(74) Attorney, Agent,
`MacPherson LLP; Samuel G. Campbell, III
`
`Morrill
`
`(57)
`
`ABSTRACT
`
`A set of haptic elements (haptels) are arranged in a grid.
`Each haptel is a haptic feedback device with linear motion
`and a touchable surface substantially perpendicular to the
`direction of motion. In a preferred embodiment, each haptel
`has a position sensor which measures the vertical position of
`the surface within its range of travel, a linear actuator which
`provides a controllable vertical bi-directional feedback
`force, and a touch location sensor on the touchable surface.
`All haptels have their sensors and effectors interfaced to a
`control processor. The touch location sensor readings are
`processed and sent to a computer, which returns the type of
`haptic response to use for each touch in progress. The
`control processor reads the position sensors, derives
`velocity, acceleration, net force and applied force
`measurements, and computes the desired force response for
`each haptel. The haptels are coordinated such that force
`feedback for a single touch is distributed across all haptels
`involved. This enables the feel of the haptic response to be
`independent of where touch is located and how many haptels
`are involved in the touch. As a touch moves across the
`device, haptels are added and removed from the coordina(cid:173)
`tion set such that the user experiences an uninterrupted
`haptic effect. Because the touch surface is comprised of a
`multiple haptels, the device can provide multiple simulta(cid:173)
`neous interactions, limited only by the size of the surface
`and the number of haptels. The size of the haptels determines
`the minimum distance between independent touches on the
`surface, but otherwise does not affect the properties of the
`device. Thus, the device is a pointing device for graphical
`user interfaces which provides dynamic haptic feedback
`under application control for multiple simultaneous interac(cid:173)
`tions.
`
`12 Claims, 9 Drawing Sheets
`
`610-..
`
`Valve Exhibit 1048
`Valve v. Immersion
`
`

`

`US 6,337,678 Bl
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,165,897 A
`5,222,895 A
`5,241,308 A
`5,412,189 A
`5,442,788 A
`5,479,528 A
`5,518,078 A
`5,576,727 A
`5,581,670 A
`5,583,478 A
`5,587,937 A
`5,617,114 A
`5,623,582 A
`5,625,576 A
`5,633,660 A
`5,643,087 A
`5,691,748 A
`5,691,898 A
`5,694,013 A
`5,694,150 A
`5,701,140 A
`5,709,219 A
`5,719,561 A
`5,734,373 A
`5,736,978 A
`5,739,811 A
`5,742,278 A
`5,767,839 A
`5,798,752 A
`5,805,140 A
`5,828,197 A
`5,831,408 A
`
`11/1992 Johnson ...................... 434/113
`6/1993 Fricke ........................ 434/113
`8/1993 Young ......................... 341/34
`5/1995 Cragun ....................... 235/379
`8/1995 Bier ........................... 395/650
`12/1995 Specter ....................... 382/115
`5/1996 Tsujioka et al.
`...... ...... .. 178/18
`11/1996 Rosenberg et al.
`......... 345/179
`12/1996 Bier et al. .................. 395/326
`12/1996 Renzi ...................... 340/407.1
`12/1996 Massie et al.
`.. ...... .... .. 364/578
`4/1997 Bier et al. .... .. ...... .... .. 345/113
`4/1997 Rosenberg ................... 395/99
`4/1997 Massie et al.
`.. ...... .... .. 364/578
`5/1997 Hansen et al. .............. 345/173
`7/1997 Marcus et al. ................ 463/38
`11/1997 Fukuzaki .................... 345/173
`11/1997 Rosenberg et al.
`......... 364/190
`12/1997 Stewart et al. .............. 318/561
`12/1997 Sigona et al. ............... 345/145
`12/1997 Rosenberg et al.
`......... 345/156
`1/1998 Chen et al. ...... .. ...... ... 128/782
`2/1998 Gonzales ............... 340/825.46
`3/1998 Rosenberg et al.
`......... 345/161
`4/1998 Hasser et al. ............... 345/173
`4/1998 Rosenberg et al.
`......... 345/161
`4/1998 Chen et al. ................. 345/156
`6/1998 Rosenberg .................. 345/161
`8/1998 Buxton et al. .............. 345/146
`9/1998 Rosenberg et al.
`......... 345/161
`10/1998 Martin et al. ............... 318/567
`11/1998 Jacobus et al. ........ 318/568.11
`
`5,844,560 A
`5,875,311 A
`5,880,411 A
`5,952,998 A
`5,999,168 A
`6,046,726 A
`6,100,874 A
`
`12/1998 Crutcher et al. ............ 345/354
`2/1999 Bertram et al. ............. 395/309
`• 3/1999 Gillespie et al. ......... 178/18.01
`• 9/1999 Clancy et al.
`.............. 345/173
`• 12/1999 Rosenberg et al.
`......... 345/161
`• 4/2000 Keyson ...................... 345/156
`• 8/2000 Schena et al. .............. 345/157
`
`OTHER PUBLICATIONS
`
`Fish, Daniel E., "Statement of Purpose" from the Applica(cid:173)
`tion for Admission to study at the Media Laboratory of the
`Massachusetts Institute of Technology (submitted 1/98).
`Fitzmaurice, George W., Buxton, William, "An Empirical
`Evaluation of Graspable User Interfaces: Towards Special(cid:173)
`ized, Space-Multiplexed Input" (1997) (pp. 43---50). CHI 97,
`Atlanta, GA.
`Fitzmaurice, George W., Ishii, Hiroshi, Buxton, William,
`"Bricks: Laying the Foundations For Graspable User Inter(cid:173)
`faces" (May 1995) (pp. 442-449). CHI '95, Denver, CO.
`Hinckley, Ken, Pausch, Randy, Proffitt, Dennis, Patten,
`James, and Kassell, Neal, "Cooperative Bimanual Action"
`(1997) (pp. 27-34). CHI 97, Atlanta, GA.
`Kabbash, Paul, Buxton, William, and Sellen, Abigail, "Two
`Handed Input In A Compound Task" (1994) (pp. 417-423).
`CHI94-4-94, Boston, MA.
`Kurtenbach, Gordon, Fitzmaurice, George, Bandel, Thomas
`and Buxton, Bill, "The Design Of A GUI Paradigm Based
`On Tablets, Two-Hands, and Transparency" (1997)(pp.
`35-42). CHI 97, Atlanta, GA.
`
`* cited by examiner
`
`

`

`U.S. Patent
`
`Jan.8,2002
`
`Sheet 1 of 9
`
`US 6,337,678 Bl
`
`... -··· .. ••• ___ t-.::::·.::::_:~~
`.--~~;::-----
`------·l
`i
`.
`
`·r----
`----- I
`
`'
`
`104c
`
`104b
`
`114
`
`102
`
`118
`
`104
`
`2006?'. •
`
`106
`
`106a
`
`108
`
`110
`
`Figure 2
`
`Figure 1
`
`e,<200
`,,,,.
`
`100a
`
`

`

`U.S. Patent
`
`Jan.8,2002
`
`Sheet 2 of 9
`
`US 6,337,678 Bl
`
`404
`
`404
`
`318
`
`316
`
`314
`
`9 3 1 2
`
`
`' ' ~~308
`!
`! 310
`306a :
`• 306
`
`402
`
`1 304a
`!
`I
`! 30
`I
`
`I 304
`
`I
`' I
`404~
`
`I
`I
`~404
`
`302
`
`Figure 4
`
`Figure 3
`
`

`

`U.S. Patent
`
`Jan.8,2002
`
`Sheet 3 of 9
`
`US 6,337,678 Bl
`
`114
`
`118
`
`2006?_,,..---- ✓
`
`402
`
`310
`
`Figure SA
`
`Figure SB
`
`

`

`U.S. Patent
`
`Jan.8,2002
`
`Sheet 4 of 9
`
`US 6,337,678 Bl
`
`608a,
`
`610
`
`606
`
`602 !
`I
`
`610
`
`Figure 6A
`
`

`

`U.S. Patent
`
`Jan.8,2002
`
`Sheet 5 of 9
`
`US 6,337,678 Bl
`
`114,
`118-
`310_j
`
`610
`
`Figure 6B
`
`

`

`U.S. Patent
`
`Jan.8,2002
`
`Sheet 6 of 9
`
`US 6,337,678 Bl
`
`r----------------
`
`1
`I
`
`: - - -308
`
`(
`
`L -
`
`- -I
`
`- - - - - - _.J - ,
`~ - - -310
`
`I
`
`_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ J _ _ _ _ _ _ _ _ _ _
`
`, _,700
`,---------------------------------------
`R3
`C1
`
`1
`
`+5
`
`R1
`
`R2
`
`-5
`
`R3
`
`+5 C2
`
`~
`
`702- - - :
`'j
`
`C1
`
`-5
`
`OUTPUT
`
`R4
`
`R5
`
`---------------------------------------
`-----------------------------------------------------
`---
`
`Figure 7
`
`

`

`U.S. Patent
`
`Jan.8,2002
`
`Sheet 7 of 9
`
`US 6,337,678 Bl
`
`, - ·800
`~----------------------------------------------L ________ _
`1/
`I
`4/ * /
`/
`I
`
`14/
`,
`
`8v
`/
`
`+5~---,
`I
`I
`I
`
`I
`I
`
`- -
`
`=IN
`L...l'-
`<IN
`810--' ___c:-- >IN
`4/ P0-P3
`,
`00
`01
`02
`03
`
`00
`
`DO
`LE
`
`830
`
`/865
`
`+SL
`
`=IN
`
`F <IN
`
`>IN
`
`P0-P3
`Q0-O3
`
`L =IN
`
`~
`
`<IN
`- - >IN
`
`P0-P3
`Q0-O3
`
`P=O t - - -
`P<O
`,--
`P>O.-
`
`P=O
`P<O
`P>O
`
`\_g70
`
`1 a?s
`00 ,___
`01 ,__
`
`-
`
`-
`
`DO
`D1
`LE
`
`t---805
`
`P=O
`P<O
`P>O
`
`I
`I
`L-,--- f
`815·,
`:
`I
`I
`820-' I-:.-:-.:-:_...!
`-,
`-:..:::-
`-
`
`~
`
`I
`I
`
`LE
`8 / 00-D7 04-07 4.,.
`,
`
`/
`
`I l /825
`00-03 *
`
`ass----
`-
`
`CLK
`
`C0-C3 4/
`
`,
`
`Rl
`
`LP
`,__ CLK
`
`C0-C3 4/
`.,
`~860
`
`_r835 840 840 840
`CLOCK~
`
`~~50 rB45
`
`I
`
`/885
`
`1Y
`
`1A
`
`2A
`8 8~
`
`1,2EN
`
`2Y
`
`+12
`~v89o
`
`~~890
`
`+12
`~v89o
`
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`~~890 I
`
`I
`
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I ---------------------------
`I r------------1 ----------
`---
`-
`
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`
`--......
`
`',
`
`-
`
`-~ )
`
`- - - - - -,- - - - - - - ----,-----
`,_ 108
`,_ 114
`
`Figure 8
`
`

`

`310(1)
`
`--
`
`-
`
`------------
`,,.500(1)
`- - - _/_ - - - I
`: 308(1 )J-1 PROXIMITY SENSOR I :
`
`...
`
`XYSENSOR
`
`I . . . - - - - - - - - - ,
`MAGNET WIRE
`
`I 116( 1)
`I
`
`I 108(1)
`I
`
`---------------::::::::::::::_ i
`
`:
`
`700(1)
`
`POSITION CIRCUIT I
`•
`•
`•
`POSITION CIRCUIT I
`
`700(N)
`
`XY INTERFACE
`
`912(N)
`
`•
`XY INTERFACE
`
`906
`
`922
`
`904
`
`I
`I
`I
`I
`
`CONTROL
`PROCESSOR
`
`I •I
`
`•1
`
`ANALOG
`INPUT
`CARD
`
`910
`
`SERIAL
`CARD
`
`SERIAL
`CARD
`
`910
`
`•
`•
`•
`
`,,.500(N)
`- _I_ -
`
`-
`
`-
`
`-
`
`I
`I
`I
`1 308(N) _/1 PROXIMITY SENSOR
`I I
`I
`
`I
`1 116(N)
`I
`I
`1 108(N)
`I
`
`I
`
`XYSENSOR
`
`MAGNET WIRE
`
`1 _ _ _ _ _ _ _ _ _ _
`
`'---"--< ACTUATOR CIRCUIT
`•
`•
`•
`I
`----1 ACTUATOR CIRCUIT
`j 1 ~4(N)\
`1
`- - - - - - - - - - - - - - - - - - - - - - - r ____________ ::~~::::::::t::~~
`
`800(N)
`
`DIGITAL
`OUTPUT
`CARD
`
`I
`
`914
`
`908
`
`900 _., I
`
`902- /
`
`Figure 9
`
`916
`
`COMPUTER
`
`d .
`r:JJ. .
`~
`~ ......
`~ = ......
`
`~
`~
`?
`~CIO
`N
`0
`0
`N
`
`'JJ. =(cid:173)~
`~ ....
`CIO
`0 ....,
`\0
`
`e
`rJ'J.
`O'I
`~
`~
`-...,l
`O'I
`-...,l
`~
`~
`i,-
`
`

`

`U.S. Patent
`
`Jan.8,2002
`
`Sheet 9 of 9
`
`US 6,337,678 Bl
`
`1000
`
`BEGIN
`
`1002
`
`READ POSITION SENSORS
`
`1004
`
`COMPUTE DERIVED MEASUREMENTS
`
`1006
`
`READXYDATA
`
`1 008
`
`EXAMINE XY DATA FOR NEW TOUCHES
`
`1010
`
`COMPUTE COLLECTIVE MEASUREMENTS
`
`1012
`
`ADD AND REMOVE HAPTELS FROM COORDINATION
`
`1014
`
`SEND TOUCH ATTRIBUTES
`
`1016
`
`RECEIVE HAPTIC COMMANDS
`
`1018
`
`COMPUTE COLLECTIVE FEEDBACK FORCE
`
`1020
`
`1022
`
`COMPUTE HAPTEL FEEDBACK FORCE
`
`COMPUTE HAPTEL RESTORING FORCE
`
`1024
`
`SET ACTUATOR DRIVERS
`
`Figure 10
`
`

`

`US 6,337,678 Bl
`
`1
`FORCE FEEDBACK COMPUTER INPUT
`AND OUTPUT DEVICE WITH
`COORDINATED HAPTIC ELEMENTS
`
`CROSS-REFERENCES
`
`This application is related to Disclosure Document No.
`431794 entitled "ACTIVE MULTI-TOUCH INPUT SUR-
`FACE (AMTIS)," having D. E. Fish as inventor. This
`disclosure document is hereby incorporated by reference
`herein, in its entirety and for all purposes.
`
`BACKGROUND
`
`1. Field of Invention
`This invention relates to computer input and output 15
`devices, specifically to those which provide force feedback,
`and to those which can be used as a pointing device for
`graphical user interfaces.
`2. Description of Prior Art
`Computers are becoming increasingly important as a
`productivity tool. They continue to improve dramatically in
`terms of computational speed, memory, storage and display.
`However, the interface between users and the computer has
`not changed significantly since the introduction of the mouse
`and the graphical user interface. The human-computer inter(cid:173)
`face must be improved for users to increase their produc(cid:173)
`tivity and take better advantage of the new capabilities
`computers provide.
`Many common computer interface operations are best
`performed with a direct manipulation interface. For
`example, when using a drawing application, it is easier for
`the user to point at the object they wish to select, rather than
`use a voice recognition interface in which they must
`describe the object they wish to select.
`Typically, direct manipulation interfaces combine a high(cid:173)
`resolution pointing device, used to move a cursor on the
`screen, with some way to initiate an action at the current
`location. For example, a mouse may employ rotary optical
`encoders to measure the distance moved, and one or more
`buttons for "clicking" on the object beneath the cursor (e.g.,
`selecting, actuating, dragging, or otherwise manipulating an
`on-screen object.).
`While this was a significant improvement over previous
`devices, such an interface does not come close to fully
`exploiting the abilities people have to manipulate objects
`with their hands. Existing devices have one or more of the
`following drawbacks:
`No Direct Mapping Between the Hand and the Display
`Direct mapping is used herein to describe the case where
`a one-to-one correspondence exists between the position of
`a cursor on a screen and the position of a user's hand, and
`also implies that there is a unique hand position for every
`cursor position. Input devices which do not move, such as
`trackballs, joysticks, the IBM TrackPoint™ and the Synap- 55
`tics TouchPad, lack such a direct mapping. No matter where
`the cursor is, the user's hand is in essentially the same
`location. A mouse also lacks a direct mapping, for at least
`two reasons. First, there is a non-linear relationship between
`the speed of the mouse and the speed of the cursor on the 60
`screen. This results in a different position depending on how
`quickly the mouse is moved from one location to another.
`Second, the mouse is often picked up and moved during use,
`particularly if the working area is limited.
`Direct mapping is important because it better leverages a 65
`user's spatial skills. Humans have a keen sense of the
`position of their hands in relationship to their body and their
`
`10
`
`2
`environment. Taking advantage of these spatial skills is
`valuable because the cognitive load placed on the user by the
`computer interface is decreased, leaving the user's attention
`available for performing work. For example, when dragging
`5 an object from one point on the screen to another, a user
`must pay close attention to a cursor's position and look for
`visual feedback indicating the cursor is positioned properly,
`in order to manipulate an on-screen object. During this
`process, the user's attention is not available for other tasks
`(e.g., reviewing files, program output, and the like). Some
`existing input devices have a direct mapping between the
`hand and the screen, such as touch screens and digitizing
`tablets. These devices suffer from other infirmities, as
`described below.
`Lack of Dynamic Haptic Feedback
`Haptic feedback is a preferable characteristic for input
`devices. The term haptic feedback as used herein means
`communicating information to a user through forces applied
`to the user's body. Typically, the position of some portion of
`20 an input device changes along at least one degree of freedom
`depending on the force applied by the user. For example,
`when pressing a button on a mouse, the button does not
`move until the applied force reaches a certain threshold, at
`which point the button moves downward with relative ease
`25 and then stops ( e.g., the sensation of "clicking" a button).
`The change in the position of the button communicates to the
`user through their sense of touch that the mouse click was
`successful. Note that a device with haptic feedback can be
`an input device (initiating an action) and an output device
`30 (giving haptic feedback indicating that the action was
`initiated) simultaneously.
`Input devices that are completely devoid of haptic
`feedback, such as membrane keyboards and touch screens,
`have not gained widespread acceptance for desktop com-
`35 puters as a result of this deficiency. Thus when using such
`input devices, users are uncertain whether a finger press was
`registered by the computer and so must pay special attention
`to visual or auditory feedback to get this confirmation. This
`decreases data entry rates, making users less productive and
`40 the computer interface less enjoyable to use.
`Mice, trackballs, joysticks, and other devices often pro(cid:173)
`vide buttons for initiating actions that provide haptic feed(cid:173)
`back. For example, the stylus used with a graphics tablet has
`a spring in its tip so the position of the pen relative to the
`45 tablet can vary depending on the applied force. However,
`such devices have the same hap tic response regardless of the
`state of the user interface. For example, if a user clicks the
`mouse on a graphical button that is disabled, the haptic
`response of the mouse button is no different from that of
`50 clicking a button that is enabled, and so is misleading to the
`user because no action will result from the click. What is
`needed is an input device which provides dynamic haptic
`feedback. Haptic feedback is termed herein as being
`dynamic to indicate that the haptic feedback can be altered
`over time (e.g. by means a software application) in order to
`provide additional information to a user.
`A number of devices having dynamic force feedback
`exist. Most of these lack a direct mapping between the hand
`and the device (e.g. force-feedback joysticks). Others have
`a direct mapping but are primarily designed for use in
`three-dimensional applications such as virtual reality or
`tele-operation. Most productive work done on computers is
`two-dimensional in nature, such as spreadsheets and page
`layout. These productivity applications would not enjoy
`significant benefits from the use of a three-dimensional input
`device. These devices have additional drawbacks, as out(cid:173)
`lined below.
`
`

`

`US 6,337,678 Bl
`
`3
`User Interaction is Encumbered or Impeded
`Many input devices encumber the user by requiring them
`to move at least a portion of the input device during use. For
`example, the time it takes to move the cursor across the
`screen with a mouse is increased because the user must
`accelerate and decelerate the mass of the mouse, in addition
`to the mass of their hand. Other input devices do not add
`inertia but impede the user in other ways. With a trackball,
`for example, multiple sweeping motions are required to
`move the cursor large distances, which is awkward and time
`consuming. With a joystick, for example, the force applied
`relates to the speed of the cursor on the screen, which may
`require the user to wait when the cursor is moving relatively
`large distances.
`Any input device which must be located and/or manipu(cid:173)
`lated before use suffers from such problems to at least a
`certain extent (e.g., mice and some force reflecting
`interfaces, among others). For example, if a person not
`currently using a computer and wants to press a graphical
`button on computer's display, they must find and grasp the 20
`mouse, move the mouse to position the cursor over the
`button, and then click the button. In contrast, a touch screen
`leaves the user unencumbered. They can reach out and press
`a graphical button on the display directly, with no interme(cid:173)
`diate steps. A touch screen, however, suffers from the 25
`previously-described infirmity of lacking haptic feedback.
`Insufficient Support for Multiple Interactions
`Most input devices, such as the mouse, trackball,joystick,
`the Synaptics TouchPad and the IBM TrackPoint™, only
`support a single interaction at a time. However, people have 30
`two hands which they are innately able to use together. Two
`single-interaction devices have been combined to provide
`two points of control, but confusion can arise because the
`correspondence between screen cursors and pointing
`devices is not apparent. Because these devices lack a direct 35
`mapping to the screen, their physical positions cannot
`resolve the correspondence between an input device and its
`cursor. Moreover, no provision is made for the interaction of
`multiple users. With a single input device, only a single user
`may "own" the device at any given time, and (given a single 40
`input device) users must take turns interacting with the
`computer. This is obviously a cumbersome and awkward
`technique when multiple users wish to work collaboratively
`on a given project.
`
`SUMMARY OF THE INVENTION
`Embodiments of the present invention overcomes con(cid:173)
`ventional limitations by providing a device having a direct
`mapping, for example, between the touching portion of a
`user's hand and the position of a cursor on a display and an
`output in the form of dynamic haptic feedback, without
`encumbering or impeding the user and allowing a large
`number of simultaneous interactions. The device provides
`direct mapping to reduce the conscious effort required for
`relatively pedestrian tasks such as interacting with a graphi(cid:173)
`cal user interface (GUI). The user's interaction with the
`device is not hampered by a need to laterally move any
`portion of the device.
`The device provides dynamic haptic feedback. Haptic
`feedback is termed herein as being dynamic to indicate that 60
`the haptic feedback can be altered over time (e.g. by means
`a software application) in order to provide additional infor(cid:173)
`mation to a user. In the previous example, a disabled button
`would have a different feel from that of an enabled button,
`allowing a user to discern that a graphical button was not 65
`enabled, using their sense of touch. The device also supports
`multiple interactions. Having more than two points of con-
`
`4
`trol is useful when multiple users collaborate at the same
`computer. Allowing a large number of interactions at once
`allows multiple users to interact with the computer simul(cid:173)
`taneously. Another benefit of having more than two points of
`5 control is the ability of a user to employ multiple fingers for
`pointing purposes, even in combination.
`Embodiments of the present invention take the form of an
`input and output device for a processor. In one embodiment,
`an input/output device has a horizontal two-dimensional
`10 area which can be touched simultaneously (e.g., with the
`hands) in multiple places. The location of each touch is
`measured and the area surrounding each touch moves ver(cid:173)
`tically and provides dynamic haptic feedback to the user.
`The device has a control processor that communicates with
`15 another processor on which software applications are
`executed. The control processor continually sends the cur(cid:173)
`rent attributes of all touches in progress, and receives
`commands which specify the type of haptic response each
`touch should exhibit.
`The touchable area is comprised of a grid of haptic
`elements, referred to herein as haptels. Haptel is used herein
`to describe a haptic feedback device with linear motion
`having a touchable surface substantially perpendicular to the
`direction of motion. A hap tic feedback device is used herein
`to describe an input and output device with a moving portion
`manipulated by a user, one or more sensors that measure the
`position and/or various derivatives of position and/or the
`forces applied to the moving portion, one or more effectors
`which can apply forces to the moving portion, and a pro(cid:173)
`cessor which measures the sensors, computes a response,
`and drives the effectors to create a range of haptic effects.
`In one embodiment, each haptel includes a position sensor
`to measure the vertical position of the surface within its
`range of travel, an electromagnetic linear actuator to provide
`a controllable vertical bi-directional feedback force, and a
`touch location sensor to measure the coordinates of a single
`touch within its bounds. Preferably, the haptel grid is cov(cid:173)
`ered by a single sheet of flexible material that protects the
`haptels and hides the grid from view.
`The haptels have their sensors and effectors interfaced to
`a control processor. The control processor measures the
`position of haptel surfaces and allows information such as
`velocity, acceleration, and applied force to be derived.
`45 Alternatively, sensors can be included in each haptel to
`provide such measurements (and others) directly. The con(cid:173)
`trol processor computes the desired feedback force for each
`haptel and drives the actuators to generate the appropriate
`forces. The haptic response of each haptel may be config-
`50 ured to be essentially arbitrary within a certain range. The
`range of available effects depends on the type of sensors
`employed, the bandwidth and precision of the sensors and
`effectors, the resolution of the analog-to-digital and digital(cid:173)
`to-analog conversion performed, the amount of available
`55 processing power and the update frequency of the control
`loop, among other factors. These tradeoffs would be appar(cid:173)
`ent to one skilled in the art of force feedback design.
`Because the touchable area is comprised of many haptels,
`each of which can function independently, the device allows
`multiple touches at once. Each haptel responds to only one
`touch at a time, so that there is a lower bound on the distance
`between two touches which do not interfere with each other.
`The worst-case value of this minimum distance is approxi(cid:173)
`mately the diagonal size of a haptel. However, in a specific
`instance the minimum distance can be substantially smaller
`depending on the locations of the two touches. Smaller
`haptels allow touches to be closer to one another.
`
`

`

`US 6,337,678 Bl
`
`6
`Not only can such device coordinate a fixed set of haptels,
`but it can also transparently add and remove haptels from the
`coordination set over time. This is necessary during "drag(cid:173)
`ging" operations in which touches move across the device.
`5 When a touch gets close to another haptel, the newly-added
`haptel is added to the coordination set. This has the effect of
`causing its surface to become flush with the haptels already
`involved in the touch. Preferably, this is done without
`affecting the feel of the touch in progress. When the touch
`10 moves far enough away from a given haptel, that haptel is
`removed from the coordination set, leaving it free to par(cid:173)
`ticipate in another touch.
`This coordination effectively makes the haptels' gridded
`nature invisible to the user and to software applications. The
`15 computer specifies the response for a touch in a declarative
`fashion, and the device ensures that this response will be
`generated regardless of where the touch falls, how many
`haptels are involved in the touch, or whether the touch
`moves. Device-specific information provided to the com-
`20 puter might include the minimum allowed distance between
`independent touches, so that the computer can separate
`controls designed for simultaneous use appropriately or give
`feedback to the user when one touch ventures too close to
`another.
`The foregoing is a summary and thus contains, by
`necessity, simplifications, generalizations and omissions of
`detail; consequently, those skilled in the art will appreciate
`that the summary is illustrative only and is not intended to
`be in any way limiting. Other aspects, inventive features,
`and advantages of the present invention, as defined solely by
`the claims, will become apparent in the non-limiting detailed
`description set forth below.
`
`5
`A typical interaction is a user pressing a graphical button
`displayed as part of a GUI. The finger touches the device,
`landing on a specific hap tel. The overall location of the touch
`is determined by the touch location sensor of the haptel in
`combination with the location of that haptel within the
`haptel grid. The touch location is communicated to a pro(cid:173)
`cessor (e.g., a computer) which discovers that a graphical
`button is "underneath" the touch, and therefore communi(cid:173)
`cates this information to the control processor to use a
`"button" haptic response for this touch. As the user presses
`down on the haptel, the control processor responds with a
`feedback force which increases as the surface is depressed
`until the position reaches a certain threshold, at which point
`the feedback force is quickly reduced. This causes the
`applied force to momentarily exceed the feedback force,
`which results in the quick downward movement of the
`haptel surface. In this way a "clicking" sensation is con(cid:173)
`veyed to the user. Preferably, the computer is continually
`informed of the state of the touch so that when the haptel
`reaches the bottom of its travel, the computer executes the
`action represented by the graphical button and displays the
`button in its activated state.
`If the graphical button is disabled, the computer has the
`control processor use a "disabled button" haptic response. In
`this response the feedback force increases with position at a
`higher rate than the "button" response with no force drop- 25
`off. This creates the sensation of an unyielding surface
`which informs the user than the action represented by the
`graphical button cannot be initiated.
`The preceding descriptions assume that each touch falls
`within the bounds of a single haptel, but this need not be the 30
`case. If the touchable area of the device is mapped to a GUI
`in which interface elements can be placed anywhere, some
`will happen to be located on the edge between two haptels
`or the vertex where four haptels meet. A touch on such a
`control is therefore likely land on more than one haptel. 35
`Such "border touches" can be transparently handled by the
`device. The first step is to merge related touches. If two
`touches appear simultaneously on adjacent haptels a short
`distance apart, the device can safely infer that the touches
`are really a single touch on the border between those two 40
`haptels. Similar inferences can be made for touches that
`appear simultaneously near the vertex of any number of
`haptels.
`Once the set of haptels is determined, the haptels are
`managed in a coordinated fashion. The center of the touch is
`computed, preferably by weighting each touch location by
`the force applied to that haptel, and then dividing by the total
`force applied to the haptels involved. Likewise, the collec(cid:173)
`tive surface position, velocity, and acceleration are
`computed, preferably by weighted average of the haptels
`involved. Other weightings are possible, including equal
`weighting of values. The applied force measurements of the
`haptels involved may be summed to compute the total force
`applied. The haptic response is then computed from these
`collective measurements in much the same way they would
`be computed for a single haptel, resulting in a collective
`feedback force. This feedback force is distributed across the
`haptels involved in the touch in proportion to the amount of
`the total applied force lands on each haptel. In addition, a
`restoring force pulls the haptels towards the collective 60
`position to prevent surfaces from drifting apart due to
`measurement errors and other factors. As a result, the total
`feedback force is effectively distributed across the haptels
`involved in the touch, and the haptel's surfaces will have
`similar position, velocity, and acceleration. This provides the 65
`illusion that a single surface was pressed, making the
`coordinated nature of the touch undetectable by the user.
`
`50
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`The present invention may be better understood, and its
`numerous objects, features, and advantages made apparent
`to those skilled in the art by referencing the accompanying
`drawings. In the drawings, related figures have the same
`number but different alphabetic suffixes.
`FIG. 1 is a schem