United States Patent [19]
`Rosenberg et al.
`
`I IIIII IIIIIII Ill llll lllll Ill llll llll lllll lllll Ill 111111111111111
`5,734,373
`Mar. 31, 1998
`
`US005734373A
`[11] Patent Number:
`[45] Date of Patent:
`
`[54] METHOD AND APPARATUS FOR
`CONTROLLING FORCE FEEDBACK
`INTERFACE SYSTEMS UTILIZING A HOST
`COMPUTER
`
`W09532459
`W09616397
`W09622591
`W09642078
`
`11/1995
`5/1996
`7/1996
`1Vl996
`
`WIPO.
`WIPO.
`WIPO.
`WIPO.
`
`[75]
`
`Inventors: Louis B. Rosenberg. Pleasanton; Adam
`C. Braun; Mike D. Levin. both of
`Sunnyvale, all of Calif.
`
`[73] Assignee: Immersion Human Interlace
`Corporation, San Jose, Calif.
`
`[21] Appl. No.: 566,282
`
`[22] Filed:
`
`Dec. 1, 1995
`
`Related U.S. Application Data
`
`[63] Continuation-in-part of Ser. No. 461,170, Jun. 5, 1995, Pat.
`No. 5,576,727, and Ser. No. 534,791, Sep. 27, 1995, said
`Ser. No. 461,170, is a continuation of Ser. No. 92,974, Jul.
`16, 1993, abandoned.
`Int. CI.6
`.............................. G06F 3/033; G09G 5/08
`[51]
`[52) U.S. CI ............................. 345/161; 345/163; 463/30;
`463/38
`[58) Field of Search ..................................... 345/156, 157.
`345/161, 163; 318/568.11, 568.25; 74/471 Y;
`463/37. 38, 30; 434/45; 395/99; 244/223
`
`[56]
`
`References Cited
`
`U.S. PATENf DOCUMENfS
`
`2,906,179
`3,490,059
`3,531,868
`3,875,488
`3,890,958
`3,944,798
`
`9/1959 Bower ...................................... 90/13.5
`1/1970 Paulsen et al. .. ......................... 73/133
`10/1970 Stevenson ................................. 33/174
`4/197 5 Crocker et al. ..... ... ...... ........... 318/648
`6/1975 Fister et al ............................... 128/2 S
`3/1976 Eaton ................................... 235/151.3
`
`(List continued on next page.)
`
`FOREIGN PATENT DOCUMENTS
`
`4-34610
`2254911
`W09502801
`W09520787
`W09520788
`
`2/1992
`10/1992
`1/1995
`8/1995
`8/1995
`
`Japan.
`United Kingdom .
`WIPO.
`WIPO.
`WIPO.
`
`OTHER PUBLICATIONS
`
`Minsky, Margaret et al.. "Feeling and Seeing: Issues in
`Force Display." ACM 1990, pp. 235-242.
`Herndon. J.N. et al., 'The State-of-the-Art Model M-2
`Maintenance System," Proceedings of the 1984 National
`Topical Meeting on Robotics and Remote Handling in
`Hostile Environments, American Nuclear Society, pp.
`59-65.
`Batter. James J. et al .. "Grope-I: A Computer Display to the
`Sense of Feel." pp. TA-4-188-TA-4-192.
`
`(List continued on next page.)
`
`Primary Examiner-Jeffery Brier
`Attome.» Agent, or Finn-Hickman Beyer & Weaver. LLP
`
`[57)
`
`ABSTRACT
`
`A method and apparatus for controlling and providing force
`feedback using an interface device manipulated by a user. A
`microprocessor is provided local to the interface device and
`reads sensor data from sensors that descnbes the position
`and/or other information about a user object moved by the
`user. such as a joystick. The microprocessor controls actua(cid:173)
`tors to provide forces on the user object and provides the
`sensor data to a host computer that is coupled to the interface
`device. The host computer sends high level host commands
`to the local microprocessor, and the microprocessor inde(cid:173)
`pendently implements a local reflex process based on the
`high level command to provide force values to the actuators
`using sensor data and other parameters. A provided host
`command protocol includes a variety of different types of
`host commands and associated command parameters. By
`providing a relatively small set of high level host commands
`and parameters which are translated into a panoply of forces,
`the protocol further shifts the computational burden from the
`host computer to the local microprocessor and allows a
`software developer to easily create force feedback applica(cid:173)
`tions.
`
`111 Claims, 21 Drawing Sheets
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 1
`
`

`

`5,734,373
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`11/1978 Jones ....................................... 318/681
`4/1979 BUJSon .................................... 340/709
`8/1980 Colston ............................... 340/365 L
`8/1983 Lam et al ................................. 434/45
`5/1984 Hayashi ............................... 73/862.04
`10/1984 Repperger ............................... 2441223
`10/1984 Davies ................................... 33/1 CC
`10/1985 Mabusth .................................... 178/18
`11/1985 Fraignier et al. .................... 73/862.04
`2/1986 Fraser et al ............................ 33/1 PT
`6/1986 Davies ................................... 33/1 CC
`7/1986 Watson ...................................... 73/514
`12/1986 Repperger et al ...................... 2441230
`111987 Shelden et al. ..................... 128/303 B
`3/1987 Iwano ...................................... 364/513
`3/1987 Herrington et al. .................... 340/710
`6/1987 Slocum .................................. 33/1 MP
`7/1987 Koontz ...................................... 33/551
`8/1987 Llndbom ................................. 4141735
`10/1987 Moriyasu ................................ 3641559
`11/1987 Grahn et al .......................... 73/862.04
`6/1988 Zanetti ................................ 128/303 B
`9/1988 Trieb et al. ............................. 364/559
`11/1988 Olson ...................................... 364/518
`12/1988 Brunnett .................................. 128/653
`111989 Miessler et al. .......................... 178/18
`111989 Cemenska et al. ....................... 60/393
`2/1989 Kendig et al ........................... 318/648
`3/1989 Hilton .................................. 73/862.04
`4/1989 Bell et al ........................... 364/571.05
`6/1989 LaBiche et al .................... 364n09.11
`7/1989 Blood ...................................... 324/208
`11/1989 Duimel ...................................... 341/20
`12/1989 Enderle et al ............................ 33/559
`1/1990 Tomelleri .................................. 33/503
`3/1990 Meenen, Jr ............................... 434/45
`3/1990 Hon ......................................... 434/262
`7/1990 Sapia .................................. 364/571.01
`7/1990 Blood ................................. 324/207.17
`7 /1990 Bell et al. .......................... 364/571.05
`10/1990 Gomiak .................................. 364/200
`10/1990 Herzog ...................................... 33/503
`10/1990 Zeller et al ............................... 33/502
`1/1991 Soderberg et al. ....................... 33/502
`1/1991 Stevens et al. ........................... 178/18
`4/1991 Greanias et al ........................... 380/25
`4/1991 Siva ...................................... 74/471 X
`8/1991 McMurtry et al ........................ 33/556
`9/1991 Behensky et al ......................... 434/45
`9/1991 Watanabe et al ................... 128/653 R
`12/1991 Davis et al ........................ 364/167.01
`2/1992 McMurtry et al ................. 364/474.03
`2/1992 Oyama .................................... 364/560
`3/1992 Oark et al .............................. 340/710
`4/1992 Mcintosh ........................... 318/568.22
`4/1992 Rosen ...................................... 200/6 A
`5/1992 Moncrief et al .................... 273/448 B
`6/1992 Mitchell et al .................... 364/474.03
`7/1992 Thomas, Jr.
`.. ............................ 341/20
`7/1992 Marinaccio et al ....................... 433n2
`7/1992 Oark ....................................... 340/710
`8/1992 Openiano ............................ 273/148 B
`8/1992 Edwards .................................. 367/127
`9/1992 Menahem .............................. 74/471 X
`9/1992 Burdea et al ............................... 414/5
`9/1992 McDonald .............................. 364/560
`1/1993 Culp .......................................... 73/510
`1/1993 Glynn ...................................... 364/566
`111993 Lang ......................................... 341/20
`2/1993 Erdman et al ..................... 364/474.05
`
`4,125,800
`4,148,014
`4,216,467
`4,398,889
`4,448,083
`4,477,043
`4,477,973
`4,550,221
`4,550,617
`4,571,834
`4,593,470
`4,601,206
`4,632,341
`4,638,798
`4,653,011
`4,654,648
`4,676,002
`4,679,331
`4,688,983
`4,703,443
`4,704,909
`4,750,487
`4,769,763
`4,787,051
`4,791,934
`4,798,919
`4,800,721
`4,803,413
`4,811,608
`4,819,195
`4,839,838
`4,849,692
`4,879,556
`4,888,877
`4,891,889
`4,907,970
`4,907,973
`4,942,545
`4,945,305
`4,945,501
`4,961,138
`4,961,267
`4,%2,591
`4,982,504
`4,983,786
`5,007,085
`5,007,300
`<i.040,306
`5.044,956
`5,050,608
`5,072,361
`5,088,046
`5,088,055
`5,095,303
`5,103,404
`5,107,080
`5,116,051
`5,126,948
`5,128,671
`5,131,844
`5,132,672
`5,139,261
`5,142,506
`5,142,931
`5,143,505
`5,148,377
`5,178,012
`5,181,181
`5,182,557
`5,184,306
`
`5,184,319
`5,185,561
`5,186,629
`5,187,874
`5,189,806
`5,204,824
`5,209,131
`5,220,260
`5,223,776
`5,228,356
`5,230,623
`5,243,266
`5,251,127
`5,251,156
`5,259,120
`5,259,894
`5,262,777
`5,264,768
`5,275,565
`5,289,273
`5,296,846
`5,351,692
`5,354,162
`5,379,663
`5,384,460
`5,389,865
`5,396,266
`5,396,267
`5,397,323
`5,402,582
`5,405,152
`5,412,880
`5,414,337
`5,417,696
`5,428,748
`5,429,140
`5,436,542
`5,436,640
`5,445,166
`5,451,924
`5,459,382
`5,467,763
`5,512,919
`5,513,100
`5,551,701
`5,570,111
`5,576,727
`5,589,854
`5,591,924
`5,623,582
`5,625,576
`5,629,594
`5,642,469
`
`2/1993 Kramer ........................ ........... 364/806
`2/1993 Good et al .............................. 318/432
`2/1993 Rohen ..................................... 434/114
`2/1993 Takahashi et al. ........................ 33/502
`3/1993 McMurtry et al. ....................... 33/503
`4/1993 Fujimaki ............................ 364/474.03
`5/1993 Baxter .................................... 73/865.8
`6/1993 Schuler ..... .......... .................... 318/561
`6/1993 Radke et al .......................... 318/568.1
`7/1993 Chuang ................................. 74/471 X
`7 /1993 Guthrie et al. ............................ 433n2
`9/1993 Kasagami et al .................... 318/568.l
`10/1993 Raab .................................. 364/413.13
`10/1993 Heier et al. ............................. 364/559
`11/1993 Chapman et al.
`........................ 33/502
`11/1993 Sampson ................................... 156/64
`11/1993 Low et al ................................. 341/20
`11/1993 Gregory et al ......................... 318/561
`1/1994 Moncrief ................................... 434/29
`2/1994 Lang ....................................... 348/121
`3/1994 Ledley .................................... 345/161
`10/1994 Dow et al .......................... 128/662.06
`10/1994 Burdea et al ............................... 414/5
`111995 Hara ...................................... 74/471 X
`1/1995 Tseng ................................. 250/231.14
`2/1995 Jacobus et al. ................. ... 318/568.11
`3/1995 Brimhall ................................. 345/161
`3/1995 Bouton .................................... 345/168
`3/1995 Taylor et al. ........................... 606/130
`4/1995 Raab ......................................... 33/503
`4/1995 Katanics et al ......................... 273/438
`5/1995 Raab ......................................... 33/503
`5/1995 Schuler ................................... 318/561
`5/1995 Kashuba et al .......................... 606/9.1
`6/1995 Davidson et al ....................... 395/275
`7/1995 Burdea et al ........................... 128n74
`7/1995 Petelin et al ............................ 318/567
`7/1995 Reeves .................................... 345/161
`8/1995 Taylor ..................................... 128/897
`9/1995 Massimino et al .................. 340/407.1
`10/1995 Jacobus et al. .................... 318/568.11
`11/1995 McMahon et al ...................... 600/201
`4/1996 Araki ...................................... 345/156
`4/1996 Parker et al ....................... 364/167.01
`9/1996 Bouton et al ............................. 463/36
`10/1996 Barrett et al. ........................... 345/157
`11/1996 Rosenberg et al. ..................... 345/179
`12/1996 Tsai ......................................... 345/161
`1/1997 Hilton ................................ 73/862.043
`4/1997 Rosenberg ................................ 395/99
`4/1997 Massie et al ........................... 364/678
`5/1997 Jacobus et al ..................... 318/568.11
`6/1997 Hannaford et al ........................ 395/99
`
`CTrHER PUBLICATIONS
`
`Gotow, J.K.., et al .. "Preception of Mechanical Properties at
`the Man-Machine Interface," IEEE 1987, pp. 688-689.
`"Proceedings of the IFIP Congress 65," International Fed(cid:173)
`eration for Information Processing. Information Processing
`1965, vol. 3, New York. May 24-29. 1965. p. 506.
`Atkinston, William D. et al. "Computing with Feeling,"
`Comput. & Graphics, vol. 2, No. 2-E, pp. 97-103.
`Noll, A. Michael. "Man-Machine Tactile Communication
`Dissertation," Polytechnic Institute of Brooklyn, Jun. 197 L
`pp. 1-88.
`Ouh-Youog. Ming, "Force Display in Molecular Docking,"
`Chapel Hill 1990, pp. 1-85.
`Ouh-youog, Ming et al., "Using a Manipulator for Force
`Display in Molecular Docking," IEEE 1988, pp. 1824-1829.
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 2
`
`

`

`5,734,373
`Page 3
`
`Wtlcer, Steven F. et al., "Development of Tactile Mice for
`Blind Access to Computers: Importance of Stimulation of
`Locus. Object Size, and Vibrotactile Display Resolution,"
`Proceedings of the Human Factors Society 35th Annual
`Meeting 1991. pp. 708-712.
`"Foot-Operated Mouse." IBM Technical Disclosure Bulle(cid:173)
`tin. Apr. 1986, vol. 28. No. 11.
`Tavkhelidze, D.S., "Kinematic Analysis of Five-Link
`Spherical Mechanisms," Mechanism and Machine Theory.
`1974, vol. 9. pp. 181-190.
`"Cursor Walso," Designer's Comer-Useful Technology for
`Your Idea File. Design News. Mar. 7, 1993, p. 63.
`"High Performance Model of the Immersion Probe." Immer(cid:173)
`sion Probe-MD™. Immersion Corporation (1994).
`Krueger. Myron W., Artificial Reality 1988. pp. 54-75.
`Adachi. Yoshitaka et al., "Sensory Evaluation of Virtual
`Haptic Push-Buttons," Technical Research Center. Suzuki
`Motor Corporation, Nov. 1994.
`Su. S. Augustine et al .. "The Virtual Panel Architecture: A
`30 Gesture Framework." IEEE 1993. pp. 387-393.
`Jacobsen, S.C. et al .. "High Performance, High Dexterity.
`Force Reflective Teleoperator II," ANS Topical Meeting on
`Robotics & Remote Systems. Albuquerque, New Mexico
`Feb. 24-27, 1991. pp. 1-10.
`Kotoku. Tetsuo et al., "Environment Modeling for the Inter(cid:173)
`active Display (EMID) Used in Telerobotic Systems," IEEE
`Nov. 3-5. 1991. pp. 99-1004.
`Bejczy, Antal K.. "The Phantom Robot: Predictive Displays
`for Teleoperation with Time Delay," IEEE 1990. pp.
`546-550.
`Buttolo, Pietro et al., "Pen-Based Force Display for Preci(cid:173)
`sion Manipulation in Virtual Environments." IEEE Mar.
`1995. pp.1-8.
`Tan. Hong Z. et al.. "Human Factors for the Design of
`Force-Reflecting Haptic Interfaces," Tan, Srinivasan, Eber(cid:173)
`man, & Chang, ASME WAM 1994. pp. 1-11.
`Ellis, R.E. et al .• "Design and Evaluation of a High-Perfor(cid:173)
`mance Prototype Planar Haptic Interface," ASME Dec. 3.
`1993, DSC-vol. 49, pp. 55-64.
`Adelstein Bernard D. et al., "A High Performance Two
`Degree-of-Freedom Kinesthetic Interface," Massachusetts
`Institute of Technology 1992, pp. 108-112.
`Colgate J. Edward et al., Implementation of Stiff Virtual
`Walls in Force-Reflecting Interfaces, Sep. 22, 1993.
`Iwata, Hiroo et al, Volume Haptization. IEEE 1993, pp.
`16-18.
`Fischer. Patrick et al.. "Specification and Design of Input
`Devices for Teleoperation," 1990.
`Burdea. Grigore et al., "Distributed Vutual Force Feed(cid:173)
`back." IEEE. May 2. 1993, pp. 25-44.
`Rosenberg, Louis B., ''The Use of Virtual Fixtures as Per(cid:173)
`ceptual Overlays to Enhance Operator Performance in
`Remote Environments," Air Force Material Command. Sep.
`1992. pp. 1-42.
`Rosenberg. Louis B .. Crew Systems Directorate Biodynam(cid:173)
`ics and Biocommunications Division Wright-Patterson. Air
`Force Material Command, Mar. 1993. pp. 1-45.
`Rosenberg, Louis B .• "Perceptual Design of a Virtual Rigid
`Surface Contact," Center for Design Research Stanford
`University, Air Force Material Command. Apr. 1993. pp.
`1-41.
`Rosenberg. Louis B. et al.. "Perceptual Decomposition of
`Virtual Haptic Surfaces," IEEE. Oct. 1993.
`
`Rosenberg. Louis B .• "Virtual Fixtures as Tools to Enhance
`Operator Performance in Telepresence Environments." SPIE
`Telemanipulator Technology. 1993.
`Rosenberg, Louis B.. ''Virtual Haptic Overlays Enhance
`Performance in Telepresence Tasks," SPIE 1994.
`Burdea, Grigore et al .. "Dextrous Telerobotics with Force
`Feedback-An Overview." Robotica 1991. vol. 9.
`Colgate, J. Edward et al .. "Implementation of Stiff Virtual
`Walls in Force-Reflecting Interfaces," 1993. pp. 1-9.
`Yarnakita, M. et al.. Tele-Virtual Reality of Dynamic
`Mechanical Model. IEEE Jul. 7-10. 1992. pp. 1103-1110.
`Adlestein, Bernard D. et al .. "Design and Implementation of
`a Force Reflecting Manipulandum for Manual Control
`Research," 1992, pp. 1-24.
`Ouh-young. Ming et al., "Force Display Performs Better
`than Visual Display in a Simple 6-D Docking Task." IEEE
`1989,pp. 1462-1466.
`Kim. Won S. et al., "Graphics Displays for Operator Aid in
`Telemanipulation," IEEE 1991. pp. 1059-1067.
`Hannaford, Blake et al., "Performance Evaluation of a
`Six-Axis Generalized Force-Reflecting Teleoperator,"
`IEEE May/Jun. 1991. vol. 21. No. 3. pp. 620-633.
`Kim. Won S. et al .. A Teleoperation Training Simulator with
`Visual and Kinesthetic Force Virtual Reality.
`Burdea, Grigore et al .. "A Portable Dextrous Master with
`Force Feedback." Presence: Teleoperators and Virtual Envi(cid:173)
`ronments. MIT Press. Jun. 1991.
`Fisher, S.S. et al .. ''Virtual Environment Display System."
`ACM Interactive 30 Graphics, Oct. 1986.
`''The Personal Digitizer™,"Immersion Human Interface
`Corporation 1994.
`"High Performance Model of the Immersion Probe." Immer(cid:173)
`sion Probe-MD™, Immersion Human Interface Corpora(cid:173)
`tion.
`30 Human Interface Tool. Immersion Probe™. Immersion
`Human Interface Corporation 1994.
`Meyer. Kenneth et al., "A Survey of Position Trackers," The
`Massachusetts Institute of Technology 1992. Presence. vol.
`1, No. 2.
`"Useful Technology for Your Idea File," Design News. Mar.
`7, 1994, p. 63.
`Smith, Geoffrey, "Call It Palpable Progress." Business
`Week, Oct. 9, 1995, pp. 93, 96.
`Repperger. D.W., "Active Force Reflection Devices in Tele(cid:173)
`operation", IEEE Control Systems, 1991. pp. 52-56.
`Winey ill, Calvin M.. "Computer Simulated Visual and
`Tactile Feedback as an Aid to Manipulator and Vehicle
`Control", Mif. 1981.
`Iwata, Hiroo. "Pen-based Hapitc Virtual Environment."
`Institute of Engineering Mechanics. U niverisity of Tsukuba.
`Japan, pp. 287-292.
`Schmult, Brian et al .. "Application Areas for a Force-Feed(cid:173)
`back Joystick." ASME 1993, DSC-vol. 49, pp. 47-54.
`McAffee, Douglas A., "Teleoperator System!felerobot
`Demonstrator: Force Reflecting Hand Controller Equipment
`Manual," JPL Jan. 1988. pp. 3-8, 11, and A-34.
`Howe, Robert D., '7ask Performance with a Dextrous
`Teleoperated Hand System," Proceedings of SPIE, Nov.
`1992, vol. 1833, pp. 1-9.
`Snow, E. et al., "Compact Force-Reflecting Hand Control(cid:173)
`ler," JPL.Apr. 1991. vol. 15, No. 3. Item No. 153.pp. l-15a.
`Tan, Hong Z et al.. "Manual Resolution of Compliance
`When Work and Force Cues are Minimized," ASME 1993,
`DSC-vol. 49, pp. 99-104.
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 3
`
`

`

`5,734,373
`Page 4
`
`Kotok:u, Tetsuo, "A Predictive Display with Force Feedback
`and its Application to Remote Manipulation System with
`Transmission Time Display," IEEE 1992, Jul. 7-10. 1992,
`pp. 239-246.
`
`Russo, Massimo Andrea. 'The Design and Implementation
`of a Three Degree--0f-Freedom Force Output Joystick."
`
`Department of Mechanical Engineering, May lL 1990. pp.
`9-40 & 96 & 97.
`Jones, L.A., et al., "A Perceptual Analysis of Stiffness,"
`Experimental Brain Research 1990, pp. 151-156.
`Hasser, Christopher John, 'Tactile Feedback for a Force-Re(cid:173)
`flecting Haptic Display," The School of Engineering. Uni(cid:173)
`versity of Dayton. Dec. 1995, pp. iii-xii & 1-96.
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 4
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 1 of 21
`
`5,734,373
`
`16
`
`21
`
`,-10
`
`/
`
`20
`
`I
`
`I
`
`,.-12
`HOST COMPUTER SYSTEM
`-------------------------------------~
`I
`18
`I
`SYSTEM
`:
`HEAR
`AUDIO OUT-
`T---1
`CLOCK
`PUT DEVICE
`:
`I
`:
`I
`HOST
`I
`-~-VIEW~
`DISPLAY
`PROCESSOR
`:
`I
`DEVICE
`-- : _________________________ :
`I
`I
`:GAME
`:PORT
`I
`; , . 25 FORCE FEEDBACK
`24
`:
`INTERFACE DEVICE
`;14
`r------- ---r-----------------------~-----,
`26
`36
`28
`:
`. - - - - - - . . . . ._ ._~
`I
`I
`LOCAL
`'
`MICRO(cid:173)
`PROCESSOR
`
`SERIA
`PORT
`
`1
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`29: ---~
`CLOCK
`
`38
`
`27
`
`ACTUATOR
`INTERFACE
`
`41
`
`SAFETY
`SWITCH
`
`I
`. . . . - - -~ - - . . . . . , I
`
`ACTUATORS
`
`:
`I
`
`~~ ... --.. , I
`
`30
`I
`__________ J
`
`--------- ------1 -------------
`
`1
`:
`
`POWER
`SUPPLY
`I
`I
`I
`I
`.. _____________ 1
`
`40
`
`!Figure 1
`
`SENSOR
`INTER·
`FACE
`
`SENSORS
`
`39
`OTHER
`INPUT
`
`34
`
`OBJECT
`
`I
`
`I I
`I I
`: FEELI I
`- - ' I
`I
`: -
`,--:=-i
`-,
`: MANIP-1
`,
`I
`L.._ _ _j __ __J_ ULATEJ
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 5
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 2 of 21
`
`5,734,373
`
`MICRO(cid:173)
`PROCESSOR
`
`26
`
`!figure2
`
`,38
`
`,,44
`' I
`
`I
`I
`: +SV
`I
`
`+SV
`
`48
`
`DAC
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- _I
`
`OUTPUT,
`
`I
`
`I
`I
`
`I
`
`,46
`--------------------------------------~---------,
`'
`c
`
`I
`
`R
`
`R
`
`R2
`
`~
`
`R
`
`56
`
`I
`I
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - _J
`
`ACTUATOR
`
`57
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 6
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 3 of 21
`
`5,734,373
`
`MICRO(cid:173)
`PROCESSOR
`
`26
`
`,38'
`~----------------------------------------
`
`1
`
`I
`l
`
`+12V
`
`DAC
`
`44
`
`+12 V
`
`\ - - - - - -
`R
`
`,--r-64
`I
`I
`I
`
`_________ _.
`
`I
`
`.____,.i ACTUATOR .,.___,
`
`30
`
`!Figure3
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 7
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 4 of 21
`
`5,734,373
`
`j/" 70
`
`72
`
`74
`
`POWER-UP HOST
`COMPUTER AND
`INTERFACE DEVICE
`
`MICROPROCESSOR
`
`ACTIVATE
`INTERFACE DEVICE
`
`HOST COMPUTER
`
`76
`
`PROCESS HOST
`APPLICATION
`
`78
`
`RECEIVE
`SENSOR DATA
`FROM MICRO(cid:173)
`PROCESSOR
`
`YES
`
`82
`
`DETERMINE LOW-LEVEL
`FORCE COMMAND IN
`ACCORDANCE WITH A
`SELECTED REFLEX
`PROCESS, SENSOR DATA,
`HOST APPLICATION, AND
`OTHER DATA
`
`SEND LOW-LEVEL
`FORCE COMMAND TO
`MICROPROCESSOR
`
`84
`
`READ RAW
`DATA FROM
`SENSORS
`
`PROCESS
`RAW DATA
`INTO SENSOR
`DATA
`
`SEND
`SENSOR
`DATA TO
`HOST
`
`91
`
`OUTPUT
`LOW-LEVEL
`FORCE COMMAND
`TO ACTUATORS
`
`94
`
`!figure 4
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 8
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 5 of 21
`
`5,734,373
`
`100...._
`
`102
`
`HOST COMPUTER
`
`PROCESS HOST
`APPLICATION
`
`POWER-UP HOST
`COMPUTER AND
`INTERFACE DEVICE
`
`MICROPROCESSOR
`
`104
`
`ACTIVATE
`INTERFACE DEVICE
`
`READ
`RAW
`DATA FROM
`SENSORS
`
`118
`
`120
`
`_....; ___ ....._ __
`
`READ SENSOR
`DATA OF USER
`OBJECT FROM
`MICRO(cid:173)
`PROCESSOR
`
`108
`
`PROCESS RAW
`DATA INTO
`SENSOR DATA
`
`STORE AND
`SEND SENSOR
`DATA TO HOST
`
`121
`
`DETERMINE
`HIGH LEVEL
`COMMAND
`
`112
`
`ISSUE HIGH LEVEL
`FORCE COMMAND TO - - - (cid:173)
`MICROPROCESSOR
`
`114
`
`~igure 5
`
`SELECT REFLEX
`PROCESS BASED
`ON HIGH LEVEL
`FORCE
`COMMAND
`
`126
`
`124
`
`DETERMINE FORCE
`COMMAND TO
`ACTUATORS IN
`ACCORDANCE WITH
`REFLEX PROCESS,
`SENSOR/OTHER DAT A,
`AND HIGH LEVEL
`FORCE COMMAND
`
`OUTPUT LOW-LEVEL
`'-------1 FORCE COMMAND
`
`128
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 9
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 6 of 21
`
`5,734,373
`
`140--........
`
`c
`'
`I r1s3
`
`.--34
`
`144
`
`' \
`,\
`-------~- ' /
`__,,,..,~:\
`\
`
`14Ba
`
`I
`
`I ,....--0
`~
`
`\
`
`\
`\
`
`.........___
`
`1
`
`.........___ - I
`--..__ E
`
`I
`
`t--153
`I .
`c
`
`:figure 6
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 10
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 7 of 21
`
`5,734,373
`
`c
`
`160~
`
`140~
`
`126
`
`164~
`
`174a
`
`168
`
`---144
`
`1'igure 7
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 11
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 8 of 21
`
`5,734,373
`
`180---..._
`
`K
`I .
`
`194
`
`.
`I
`i---34
`I .
`I .
`
`189b
`
`.......
`M
`
`:Figure 8
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 12
`
`

`

`300,
`
`1302
`
`RATE CONTROL COMMANDS
`
`EXAM
`
`1304
`
`1310
`
`,METERS
`
`SUBCL
`
`RESTORING
`
`RESTORING
`SPRING
`SLUGGISH
`STICK
`
`UNSTABLE
`
`VECTOR
`
`VIBRATION
`
`WOBBLE
`
`BUTTON
`
`JOLT
`
`R_FORCE (MAGNITUDE, DURATION,,
`30p__./
`308
`STYLE, DEADBAND)
`'---R_SPRING (COEFFICIENT, DURATION,
`STYLE, DEADBAND)
`
`SLUG (COEFFICIENT, DURATION, STYLE)
`
`CONDITION
`
`CONDITION
`
`CONDITION
`
`UNSTABLE (MAGNITUDE, DURATION, STYLE)
`
`CONDITION
`
`V_FORCE (MAGNITUDE, DURATION,
`DIRECTION)
`V_VIBRATE (MAGNITUDE, DURATION, DIRECTION,
`FREQUENCY, STYLE)
`
`WOBBLE (MAGNITUDE, DURATION, STYLE}
`
`B_FORCE (MAGNITUDE, DURATION, DIRECTION,
`BUTTON, AUTOFIRE)
`
`JOLT (MAGNITUDE, DURATION, DIRECTION)
`
`OVERLAY
`
`OVERLAY
`
`OVERLAY
`
`OVERLAY/
`CONDITION
`
`OVERLAY
`
`:Jigure9
`
`~ •
`00 •
`
`~ ....... a
`
`f !"I
`
`~
`i,..,.
`"'
`
`i,..,. !
`
`g:
`a
`
`l,C
`~
`N
`i,..,.
`
`01
`.... .......
`
`~
`~
`~
`~
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 13
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 10 of 21
`
`5,734,373
`
`312--x
`
`FORCE
`
`---
`
`r316
`
`DISPLACEMENT
`
`316__/
`
`I
`
`I _.
`
`I
`
`F
`
`!figure10a
`
`314-.,,.,,._
`
`F
`
`FORCE
`
`,-316
`
`DISPLACEMENT
`
`!figure106
`
`,316
`
`FORCE
`
`--
`
`F
`
`I
`I
`
`~R
`
`DISPLACEMENT
`
`!figure 10c
`
`R_.:
`
`I
`
`I
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 14
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 11 of 21
`
`5,734,373
`
`FORCE
`
`320--,.._
`
`DISPLACEMENT
`
`:Jigurella
`
`FORCE
`
`322~
`
`O
`
`DISPLACEMENT
`
`:Jigure116
`
`324~
`
`FORCE
`
`DISPLACEMENT
`
`:Jigure11c
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 15
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 12 of 21
`
`5,734,373
`
`y
`
`326~
`
`x
`
`:Jigure12
`
`FORCE
`
`,-328
`
`TIME
`
`:Jigure13a
`
`FORCE
`
`,-330
`
`TIME
`
`:Jigure136
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 16
`
`

`

`332,
`
`POSITION CONTROL COMMANDS
`
`~336
`
`~334
`
`I
`
`FORCE
`
`VECTOR
`
`GROOVE
`
`DIVOT
`
`TEXTURE
`
`BARRIER
`
`FIELD
`
`PADDLE
`
`BUTTON
`
`EXAMPLE OF COMMAND AND PARAMETERS
`,338
`/
`,,....-340
`
`-----
`,
`
`;~
`,
`
`-- ·cf
`
`P _FORCE (MA
`
`~
`
`ITUDE, DIRECT N)
`
`GROOVE (MAGNITUDE, STYLE, SNAP-DISTANCE)
`
`DIVOT (MAGNITUDE, STYLE, SNAP-DISTANCE)
`
`TEXTURE (MAGNITUDE, DENSITY, STYLE)
`
`X_BARRIER (MAGNITUDE, LOCATION, SNAP-DISTANCE)
`
`FIELD (MAGNITUDE, ORIGIN, SENSE)
`
`PADDLE (MASS, GRAVITY, COMPLIANCE, SENSE)
`
`BUTTON (MAGNITUDE, DIRECTION, BUTTON, AUTOFIRE)
`
`:Jigure14
`
`0 •
`rJJ.
`•
`~ = """"
`t'O a
`
`:::
`~
`~ .....
`--.....
`!
`
`rr.i c:r
`!l .....
`
`~
`~
`N .....
`
`fJI
`....
`....J
`~ ....
`~
`....J
`~
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 17
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 14 of 21
`
`5,734,373
`
`342~
`
`oa ...... :
`
`s
`
`FORCE
`
`·~
`
`I
`
`s
`
`DISPLACEMENT
`
`:figure15
`
`FORCE
`
`,-,-346
`
`,/348
`
`. .,....__
`
`I
`I
`THICKNESS
`
`DISPLACEMENT
`
`:figure16
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 18
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 15 of 21
`
`5,734,373
`
`?.Q 352'@+
`
`350~
`
`350
`
`350
`
`350
`
`350
`
`350
`
`352~t
`350"),.._: __ _.wd ___ _,
`
`350
`
`352
`
`g
`
`•
`
`!fig.17a
`
`!fig.176
`
`!fig.17c
`
`!fig.17d
`
`!fig.17e
`
`!fig.17f
`
`!fig.17g
`
`!fig.17/i
`
`!fig.17i
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 19
`
`

`

`REPORTING
`~~~~~~--,•-I PROCESS
`
`SENSOR, TIMING DATA
`
`4
`
`REPORTING
`PARAMETERS
`
`386)
`
`3881
`
`384-
`
`COMMAND
`PROCESS
`
`I
`
`t
`
`INCOMING COMMANDS
`
`STATUS
`UPDATE
`PROCESS
`...
`
`REPORTING
`PARAMETERS
`
`FORCE ALGORITHM
`COMPUTATION
`AND ACTUATOR
`CONTROL PROCESS
`
`t
`
`SENSOR AND
`TIMING DATA
`
`'
`
`l
`
`3821
`
`1
`
`HOST
`COMMUNICATION
`BACKGROUND
`PROCESS
`
`~
`
`w
`
`FORCE FEEDBACK CONTROL PARAMETERS
`----------------------------------------------------~----~
`w
`
`HOST
`COMPUTER
`
`t---12
`
`:F,gure18
`
`30--4 ACTUATORS
`
`~ • rJJ •
`~ ....... a
`
`~
`~
`w
`!""'
`
`"""" i
`
`g:
`!
`"""" ~
`~
`N
`""""
`
`(.II
`':...I
`~
`ii.
`~
`~
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 20
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 17 of 21
`
`5,734,373
`
`390
`
`¥"384
`
`391
`
`ESTABLISH
`COMMUNICATION
`LINK
`
`SEND INTERFACE
`392 DEVICE INFORMATION
`TO HOST UPON
`REQUEST
`
`!Figure 19
`
`394
`
`RECEIVE COMMANDS
`AND SET
`APPROPRIATE
`REPORTING
`PARAMETERS
`
`SET FORCE
`PARAMETERS
`~-1111'1 BASED ON FORCE
`COMMAND AND
`COMMAND
`PARAMETERS
`
`402
`
`YES
`
`400
`
`SET APPROPRIATE
`REPORTING
`PARAMETERS
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 21
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 18 of 21
`
`5,734,373
`
`410
`
`!figure 20
`
`412
`
`EXAMINE REPORTING
`AND FORCE
`PARAMETERS
`
`J,t""'386
`
`414
`
`DETERMINE WHICH
`SENSORS TO BE
`READ FROM BOTH
`TYPES OF
`PARAMETERS
`
`NO
`
`YES
`
`420
`
`COMPUTE
`VELOCITY AND/OR
`ACCELERATION IF
`APPROPRIATE
`
`418
`
`COMPUTE
`VELOCITY AND/OR
`ACCELERATION
`USING SENSOR
`READINGS AND
`TIME DATA
`
`422
`
`STORE SENSOR
`DATA AND TIME
`DATA
`
`424 WAIT IF NECESSARY .,._ ______ ____.
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 22
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 19 of 21
`
`5,734,373
`
`QUERY
`
`STREAM
`
`YES
`
`YES
`
`REPORT DATA
`ACCORDING TO
`REPORTING
`PARAMETERS AND
`REQUEST
`
`440
`
`REPORT DATA
`ACCORDING TO
`REPORTING
`PARAMETERS
`
`:Figure 21
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 23
`
`

`

`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 20 of 21
`
`5,734,373
`
`/388
`
`YES
`
`SEND ZERO
`FORCE TO
`ACTUATORS
`
`470
`
`SET ERROR FLAG
`
`4 72
`
`SEND FORCES
`TO APPROPRIATE - - (cid:173)
`ACTUATORS
`
`476
`
`WAIT IF NECESSARY
`
`474
`
`:Figure 22
`
`SELECT AXIS AND SET
`TOTAL FORCE = 0
`
`CHECK FORCE
`PARAMETERS TO
`DETERMINE NEXT
`FORCE PROCESS TO
`EXECUTE AND
`COMPUTE NEXT FORCE
`
`456
`
`ADD FORCE TO
`TOTAL FORCE
`
`462 STORETOTALFORCE
`FOR THIS AXIS
`
`466
`
`LIMIT TOTAL FORCE FOR
`EACH AXIS DEPENDING
`ON HARDWARE
`CONSTRAINTS AND
`ERROR FLAG
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 24
`
`

`

`PARAMETERS
`, 486b
`486, a
`~(+) Bx(-)
`'
`
`484)
`
`308 I
`
`SLUGGISH (50, X, 81)
`
`\ '50
`
`50
`
`SLUGGISH (90, X(+), U
`
`SPRING (65, X, Bl, 85)
`
`90
`
`90
`
`50
`
`50
`
`480
`
`~ SLUGGISH FORCE
`
`RESTORING SPRING
`FORCE PARAMETERS
`
`482
`)
`
`Kx(+) Kx(-) Ky(+) Ky(-) DBx DBv
`
`0
`
`0
`
`0
`
`0
`
`0
`a
`
`0
`
`0
`
`85
`
`0
`
`0
`
`0
`
`a
`0
`
`a
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`65
`
`65
`
`!figure23
`
`~ • rJJ •
`~ = ;-a
`
`~
`
`~
`~
`"""' ~
`~ I
`
`00 =-!
`
`N
`~
`
`~
`N
`"""'
`
`(II
`~ ....J
`~
`~
`~
`~
`
`APPLE INC.
`EXHIBIT 1109 - PAGE 25
`
`

`

`5,734,373
`
`1
`METHOD AND APPARATUS FOR
`CONTROLLING FORCE FEEDBACK
`INTERFACE SYSTEMS UTILIZING A BOST
`COMPUTER
`
`CROSS REFERENCE TO REL,,ITED
`APPLICATIONS
`
`This application is a continuation-in-part of parent patent
`application Ser. No. 08/461,170. now U.S. Pat. No. 5,576,
`727. filed Jun. 5, 1995 on behalf of Louis B. Rosenberg et
`al., entitled, '"Three-Dimensional Mechanical Mouse,"
`which is a continuation of application Ser. No. 08/092,974.
`filed Jul. 16, 1993, now abandoned; and co-pending parent
`patent application Ser. No. 08/534,791, filed Sep. 27, 1995,
`on behalf of Louis B. Rosenberg. entitled, "Method and
`Apparatus for Controlling Human-Computer Interface Sys(cid:173)
`tems Providing Force Feedback," both assigned to the
`assignee of this present application, and both of which are
`incorporated by reference herein.
`
`BACKGROUND OF THE INVENTION
`
`The present invention relates generally to interface
`devices between humans and computers, and more particu(cid:173)
`larly to computer interface devices that provide force feed(cid:173)
`back to the user.
`Computer systems are used extensively in many different
`industries to implement computer controlled simulations,
`games, and other application programs. More particularly,
`these types of games and simulations are very popular with
`the mass market of home consumers. A computer system
`typically displays a visual environment to a user on a display
`screen or other visual output device. Users can interact with
`the displayed environment to play a game, experience a
`simulation or "virtual reality" environment, or otherwise
`influence events or images depicted on the screen. Such user
`interaction can be implemented through the use of a human(cid:173)
`computer interface device, such as a joystick. 'joypad"
`button controller. mouse. trackball, stylus and tablet, or the
`like, that is connected to the computer system controlling the
`displayed environment. The computer updates the simula(cid:173)
`tion or game in response to the user's manipulation of an
`object such as a joystick handle or mouse, and provides
`feedback to the user utilizing the display screen and,
`typically. audio speakers.
`In some interface devices, tactile ("haptic") feedback is
`also provided to the user, more generally known as "force
`feedback" These types of interface devices can provide
`physical sensations to the user manipulating the object of the
`interface