United States Patent
`
`[191
`
`Rosenberg et al.
`
`[11]
`
`[45]
`
`Patent Number:
`
`Date of Patent:
`
`5,734,373
`
`Mar. 31, 1998
`
`USOO5734373A
`
`[54]
`
`METHOD AND APPARATUS FOR
`CONTROLLING FORCE FEEDBACK
`INTERFACE SYSTEMS UTILIZING A HOST
`COMPUTER
`
`WO9532459
`W0961 6397
`WO962259l
`WO9642078
`
`1 1/1995
`5/1996
`7/1 996
`12/1996
`
`WIPO .
`WIPO .
`WIPO .
`WIPO .
`
`Inventors: Louis B. Rosenberg. Pleasanton; Adam
`C. Bmnn; Mike D. Levin. both of
`Sunnyvale, all of Calif.
`
`Assignee:
`
`Immersion Human Interface
`Corporation. San Jose. Calif.
`
`Appl. No.: 566,282
`
`Filed:
`
`Dec. 1, 1995
`
`Related U.S. Application Data
`
`Continuation-in-part of Ser. No. 461,170, Jun. 5, 1995, Pat.
`No. 5,576,727, and SE1’. No. 534.791, Sep. 27, 1995, said
`Ser. No. 461,170, is a continuation of Ser. No. 92,974, Jul.
`16, 1993, abandoned.
`
`Int. Cl.“ .............................. G061? 3/033: GO9G 5/08
`U.S. Cl. ............................ 345/161; 345/163; 463/30;
`463/38
`Field of Search ..................................... 345/156. 157.
`345/161. 163; 318/568.11. 568.25: 74/471 Y;
`463/37. 38. 30; 434/45; 395/99; 244/223
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`Bower ................
`Paulsen et al.
`Stevenson ......
`Crocker et al.
`Fister et al.
`Eaton ............
`
`9/1959
`1/1970
`10/1970
`4/1975
`6/1975
`3/1976
`
`2,906,179
`3,490,059
`3,531,868
`3,875,488
`3,890,958
`3,944,798
`
`90/13.5
`73/133
`33/174
`318/648
`128/2 8
`.. 235/151.3
`
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`4-34610
`2/1992
`2254911
`10/1992
`WO95(Y2801
`1/1995
`8/1995
`W09S20787
`8/1995
`WO9520788
`
`Japan .
`United Kingdom .
`WIPO .
`WIPO .
`WIPO .
`
`OTHER PUBLICALFIONS
`
`Minsky. Margaret et a.l.. “Feeling and Seeing: Issues in
`Force Display.” ACM 1990. pp. 235-242.
`Hernclon. J.N. et al.. ‘The State—of-the—Art Model M-2
`Maintenance System." Proceedings of the 1984 National
`Topical Meeting on Robotics and Remote Handling in
`Hostile Environments. American Nuclear Society. pp.
`59-65.
`
`Batter. James J. et al.. “Grope-1: A Computer Display to the
`Sense of Feel.” pp. TA—4—188—TA-4-192.
`
`(List continued on next page.)
`
`Primary Eramimn-—Jeffery Brier
`Attorney, Agent, or Firm—Hickman Beyer & Weaver. LLP
`
`[57]
`
`ABSTRACT
`
`A method and apparatus for controlling and providing force
`feedback using an interface device manipulated by a user. A
`microprocessor is provided local to the interface device and
`reads sensor data from sensors that describes the position
`and/or other information about a user object moved by the
`user. such as ajoystick. The microprocessor controls actua-
`tors to provide forces on the user object and provides the
`sensor data to a host computer mat is coupled to the interface
`device. The host computer sends high level host commands
`to the local microprocessor. and the microprocessor inde-
`pendently implements a local reflex process based on the
`high level command to provide force values to the actuators
`using sensor data and other parameters. A provided host
`command protocol includes a variety of difierent types of
`host commands and associated command parameters. By
`providing a relatively small set of high level host commands
`and parameters which are translated into a panoply of forces.
`the protocol further shifts the computational burden from the
`host computer to the local microprocessor and allows a
`software developer to easily create force feedback applica-
`tions.
`
`111 Claims, 21 Drawing Sheets
`
`FOR&FElDIADK
`wrt.n.=.A.-2.2-.E1rIc.=.....'f...
`
`~
`
`‘-
`(u.
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0001
`
`

`
`5,734,373
`Page 2
`
`5,184,319
`5,185,561
`5,135,529
`5,187,874
`5,189,806
`5,204,824
`5,209,131
`5,220,260
`5,223,776
`5,228,356
`5,230,523
`
`.
`
`..
`
`.
`
`-'
`
`~
`
`-
`
`‘
`
`.
`
`~
`
`~
`
`U.S. PATENT DOCUMENTS
`
`.
`
`..
`
`411
`
`.
`
`11/1978 Jones ......
`4,125,800
`4/1979 Burson
`4,148,014
`8/1980 Colston ..
`4,216,467
`8/1983 Lam elal
`4,398,889
`5/1984 Hayashi
`.
`4,448,083
`10/1934 Repperger _
`4,477,043
`4,477,973 10/1934 Davies .... ..
`4550,22}
`10/1935
`4550_517
`11/1935
`4,571,334
`2/1936
`_,
`gggjggg
`$7133;
`4,632,341
`12/1986
`4,633,793
`1/1937 Shelden el a1.
`4,653,011
`3/1937 Iwano ...............
`4,654,643
`3/1937 Henzington el al.
`4,676,002
`6/1987 Slocum
`4,679,331
`7/1937 Koontz ..
`4,633,933
`3/1937 Lindbam
`477031443
`10,1907 Mmmu _____1
`4,704,909
`11/1937 Grahnctal.
`4,750,437
`6/1933 Zaneni
`4,769,763
`9/1933 Triebet :11.
`4,737,051
`11/1933 015011
`4,791,934 12/1933 Bnmnett
`.
`4,793,919
`1/1939 Miessleret a1.
`4711001721
`1,1909 Cemm11ae1a1_
`4711031113
`2,1939 14011010 41 31
`41011008
`311989 11111011
`4,319,195
`411989 Be“ 01 a1_ _
`4139111311
`0,1909 L311. 110
`411491099
`7,1909 010:1’
`4,379,556 “/1989 Dunne]
`4,333,377
`12/1939 Endefle et a1.
`4,391,339
`1/1990 Tomelleri
`4,907,970
`3/1990 Meenen,Jr,
`4,907,973
`3/1990 Hon
`4,942,545
`7/1990 Sapia.
`419451305
`7,1990 B1004 _________._
`4945501
`7,1990 190110191 _
`4_901_130
`10,1990 001111011
`4,961,267
`10/1990 Herzog
`4,962,591
`10/1990 216116161 al.
`419121104
`1,1991 50041041113141.
`4,933,736
`1/1991 snevens etal.
`..
`5100790115
`4,1991 016110133 01 31”
`5,007,300
`4/1991 Siva
`5,040,306
`3/1991 McMnnIy 61.3.1‘.
`5.044.956
`9/1991 Behensky et a1.
`5,050,000
`9,1991 wa1m11ee1a1_
`5072301 12,1991 030341111
`5,088,046
`2/1992
`5,033,055
`2/1992
`.
`5,095,303
`3/1992
`5,103,404
`471902 Mcmmh _
`5110710110
`4,1902 Rose“ 11111111_.
`5,115,051
`5/1992 Moncfief at 31
`5,l26,948
`6/1992 Mitchell et al.
`5,128,671
`7/1992 Thomas, Jr.
`5,131,344
`7/1992 Marinaccio etal.
`5,132,672
`7/1992
`51130261
`871992
`1
`I
`51142506
`s/1992
`5,142,931
`9/1992
`5,143,505
`9/1992
`5545577
`971992
`5,173,012
`1/1993
`5,131,131
`1/1993
`5,132,557
`1/1993
`5,184,306
`2/1993
`
`.
`......
`
`,
`
`.
`
`__
`
`.
`--
`
`.
`
`318/681
`340/709
`340/365 L
`434/45
`. 73/862.04
`244/223
`33/1 cc
`____._ 173/13
`1 73/852104
`33/1 PT
`~~~~~-—_3=;;,;~;$;
`244/230
`123/30313
`.364/513
`. 340/710
`33/IMP
`33/551
`.414n35
`_
`. 73/362.04
`. 123/30313
`.364/559
`.364/518
`.123/653
`173/13
`00,393
`1 310,040
`__ 710062104
`364/57105
`304,709.
`324,100
`341,20
`33/559
`33/503
`.. 434/45
`.434/262
`364/571.01
`194,107.17
`004,571_05
`_304,200
`33/503
`33/502
`39,502
`173/13
`3110,25
`74/471x
`33/556
`434/45
`1 121905311
`304710701
`364/474.03
`.. 364/560
`.. 340/710
`_ 3187568722
`20076 A
`, 2731443 13
`364/474.03
`341/20
`433/72
`.. 340/710
`2737148 B
`__ 357/127
`74/471 X
`414/5
`3547559
`73/510
`.. 364/566
`341/20
`364/474.05
`
`364/8016
`318/432
`434/114
`.. 33/SW
`.. 33/503
`364/474.03
`73/865.8
`318/561
`1 318/5681
`.. 74/471 X
`.. 433/72
`-3:255:31;
`3.91559
`------ 155754
`-- 54190
`345551
`-- 43409
`3457121
`3457151
`123/662.06
`--------‘“4’5
`-7474717‘
`250723‘-14
`3”‘’555“
`545715‘
`-3457155
`606/130
`33/503
`273/433
`.. 33/S03
`.313/561
`.. 606/91
`. 395/275
`
`2/1993 Kramer
`2/1993 Good et :11,
`2/1993 R01,“ ______4
`2/1993 Takahashi ct al.
`3/1993 McMu1try el al.
`4/1993 Fujimaki
`5/1993 Baxter
`6/1993 Schuler ..
`6/1993 Radke etal.
`7/1993 Chuang
`.
`7/1993 Gnthne 61 a1.
`13719;; “$53” 5‘ 31-
`5,251,156 10/1993
`2
`.
`53591190 “"993
`5»259~394 “"993
`5/751777 “"995
`53543755 “"993
`5375555
`"1994
`5939373
`271994
`5395/345
`371994
`5,351,692 10/1994
`5554452 “V1994
`5-379553
`171995
`5=354=45°
`‘"995
`5339555
`971995 7‘°°"“‘°‘
`5395355
`371995 95""'"“
`5/395357
`371995 B5540“
`.
`5397323
`3/1995 Taylor et al.
`5,402,532
`4/1995 Raab_................
`5,405,152
`4/1995 Katanrcsetal.
`5,412,880
`5/1995 Raab .........
`5,414,337
`5/1995 Schnler
`5,417,696
`5/1995 Kaslmba et a1.
`5,428,748
`6/1995 Davidsonetal.
`51499114“
`7"995 B“""f’“"”"
`5435542
`7’1995 9551"‘ 5‘ "14 -
`54551549
`7”995 R°“’°‘ ~
`514451155
`571995
`.
`Z“
`5451994
`971995
`5,459,332 10/1995 Jacobns 1-1 :11.
`5,467,763
`11/1995 M61/1ahon etal.
`5,513,919
`4/1996 Arakr ...........
`551 /19°
`471995
`4
`5551701
`971995
`5,570,111
`10/1996 Bmettetal.
`5575-797
`1171995 R°"'°“"°'E °‘ "1
`5,539,354 12/1996 Ts_a1 .......
`5591974
`171997 H‘“°“
`5-5231557
`477997 R°“'."’°‘3
`51595575
`4/1997 M5-““‘°“‘1—
`..
`5,629,594
`5/1997 Jacobus et a1.
`..
`5,642,469
`6/1997 Hannafordetal.
`OTHER PUBLICATIONS
`-
`M
`-
`-
`Gotow. ].K.. et al.. Preceptron of Mechamcal Propemcs at
`the Man-Machine Interface," IEEE 1987. pp. 688-689.
`“Proceedings of the IFIP Congress 65.” International Fed-
`eration for Information Processing. Information Processing
`1965. vol. 3. New York. May 24-29. 1965. p. 506.
`1111011131011. 0711113111 D_ 61 311 ..C0mp01,_1ng W101 F6811“...
`.
`Cornput. & Graphrcs. vol. 2. No. 2-E. pp. 97-103.
`N011. A. Michael. “Man—Mach1'ne Tactile Communication
`Dissertation.” Polytechnic Institute of Brooklyn. Jun. 1971,
`pp. 1-88.
`0uh—Young. Ming. “Force Display in Molecular Docking.”
`.
`539595174471 1990* PP’ H45‘
`Ouh-young. Ming et al.. “Using a Manipulator for Foroc
`Display in Molecular Docking.” IEEE 1988. pp. 1824-1829.
`
`-
`
`3
`
`179597
`3407407-1
`313/563.11
`600/201
`114345/156
`7157-91
`453735
`. 345/157
`- 545’179
`345/161
`79352-943
`395’99
`~35“’573
`313/563.11
`............. 395/99
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0002
`
`

`
`5,734,373
`Page 3
`
`Wilcer. Steven F. et al.. “Development of Tactile Mice for
`Blind Access to Computers: Importance of Stimulation of
`Locus. Object Size. and Vibrotactile Display Resolution.”
`Proceedings of the Human Factors Society 35th Annual
`Meeting 1991. pp. 708-712.
`“Foot-Operated Mouse.” IBM Technical Disclosure Bulle-
`tin. Apr. 1986. vol. 28. No. 11.
`Tavkhelidze. D.S.. “Kinematic Analysis of Five—Link
`Spherical Mechanisms.” Mechanism and Machine Theory.
`1974. vol. 9. pp. 181-190.
`“Cursor Walso." Designer’s Coma-Useful Technology for
`Your Idea File. Design News. Mar. 7. 1993. p. 63.
`“High Performance Model of the Immersion Probe Immer-
`sion Probe-MDW. Immersion Corporation (1994).
`Krueger. Myron W.. Artificial Reality 1988. pp. 54-75.
`Adachi. Yoshitalra et al.. “Sensory Evaluation of Virtual
`Haptic Push-Buttons.” Technical Research Center. Suzuki
`Motor Corporation. Nov. 1994.
`Su. S. Augustine et al.. "The Virtual Panel Architecture: A
`3D Gesture Framework." IEEE 1993. pp. 387-393.
`Jacobsen. S.C. et al.. “High Performance. High Dexterity.
`Force Reflective Teleoperator II." ANS Topical Meeting on
`Robotics & Remote Systems. Albuquerque. New Mexico
`Feb. 24-27. 1991. pp. 1-10.
`Kotoku. Tetsuo et al.. "Environment Modeling for the Inter-
`active Display (EMID) Used in Telerobotic Systems." IEEE
`Nov. 3-5. 1991. pp. 99-1004.
`Bejczy. Antal K.. ‘The Phantom Robot: Predictive Displays
`for Teleoperation with Time Delay.” IEEE 1990. pp.
`546-550.
`Buttolo. Pietro et al.. “Pen—Based Force Display for Preci-
`sion Manipulation in Virtual Environments.” IEEE Mar.
`1995. pp.1-8.
`Tan. Hong Z. et al.. “Human Factors for the Design of
`Force-Reflecting Haptic Interfaces." Tan. Srinivasan. Eber-
`man. & Chang. ASME WAM 1994. pp. 1-11.
`Ellis. RE et aL. “Design and Evaluation of a High-Perfor-
`mance Prototype Planar Haptic Interface.” ASME Dec. 3.
`1993. DSC-vol. 49. pp. 55-64.
`Adelstein Bernard D. et al.. “A High Performance 'I‘wo
`Degree-of-Freedom Kinesthetic Interface." Massachusetts
`Institute of Technology 1992. pp. 108-112.
`Colgate J. Edward et al.. Implementation of Stiff Virtual
`Walls in Force—Refiecti.ng Interfaces. Sep. 22. 1993.
`Iwata. Hiroo et al. Volume Haptization. IEEE 1993. pp.
`16-18.
`
`Fischer. Patrick et al.. “Specification and Design of Input
`Devices for Teleoperation.” 1990.
`Burdea. Grigore et al.. “Distributed Virtual Force Feed-
`back.” IEEE. May 2. 1993. pp. 25-44.
`Rosenberg. Louis B.. ‘The Use of Virtual Fixtures as Per-
`ceptual Overlays to Enhance Operator Perfornzmnce in
`Remote Environments." Air Force Material Command. Sep.
`1992. pp. 1-42.
`Rosenberg. Louis B.. Crew Systems Directorate Biodynam-
`ics and Biocommunications Division Wright-Patterson. Air
`Force Material Command. Mar. 1993. pp. 1-45.
`Rosenberg. Louis B.. “Perceptual Design of a Virtual Rigid
`Surface Contac.” Center for Design Research Stanford
`University. Air Force Material Command. Apr. 1993. pp.
`1-41.
`Rosenberg. Louis B. et al.. “Perceptual Decomposition of
`Virtual Haptic Surfaces.” IEEE. Oct. 1993.
`
`Rosenberg. Louis B.. “Virtual Fixtures as Tools to Enhance
`Operator Performance in Telepresence Environments.” SPIE
`Telemanipulator Technology. 1993.
`Rosenberg. Louis B.. “Virtual Haptic Overlays Enhance
`Performance in Telepresence Tasks.” SPIE 1994.
`Burdea. Grigore et al.. “Dextrous Telerobotics with Force
`Feedback—An Overview." Robotica 1991. vol. 9.
`Colgate. J. Edward et al.. “I.mplernentation of Stiff Virtual
`Walls in Force-Reflecting Interfaces." 1993. pp. 1-9.
`Yamakita. M. et al.. Te1e—Virtual Reality of Dynamic
`Mechanical Model. IEE Jul. 7-10. 1992. pp. 1103-1110.
`Adlestein. Bernard D. et al.. “Design and Implementation of
`a Force Reflecting Manipulandum for Manual Control
`Research.” 1992. pp. 1-24.
`Ouh-young. Ming ct al.. “Force Display Performs Better
`than Visual Display in a Simple 6-D Docking Task.” IEEE
`1989. pp. 1462-1466.
`Kim. Won S. et al.. "Graphics Displays for Operator Aid in
`Telemanipulation." IEEE 1991. pp. 1059-1067.
`Hannaford. Blake et al.. “Performance Evaluation of a
`Six—Axis Generalized Force-Refleaing Teleoperator."
`IEEE May/Jun. 1991. vol. 21. No. 3. pp. 620-633.
`Kim. Won S. et al.. ATeleopei-ation Training Simulator with
`Visual and Kinesthetic Force Virtual Reality.
`Btndea. Grigore et al.. “A Portable Dextrous Master with
`Force Feedback.” Presence: Teleoperators and Virtual Envi-
`ronments. MIT Press. Jun. 1991.
`Fisher. S.S. et al.. “Virtual Environment Display System”
`ACM Interactive 3D Graphics. Oct. 1986.
`‘The Personal DigitizcrW."Irnmersion Human Interface
`Corporation 1994.
`“High Performance Model of the Immersion Probe." Immer-
`sion Probe-MDTM. Immersion Human Interface Corpora-
`tion.
`3D Human Interface Tool. Immersion Probem. Immersion
`Human Interface Corporation 1994.
`Meyer. Kenneth et al.. “A Survey of Position Trackers.” The
`Massachusetts Institute of Technology 1992. Presence. vol.
`1. No. 2.
`“Useful Technology for Your Idea File." Design News. Mar.
`7. 1994. p. 63.
`Smith. Geoflrey. “Call It Palpable Progress." Business
`Week. Oct. 9. 1995. pp. 93. 96.
`Repperger. D.W.. “Active Force Reflection Devices in Tele-
`operation”, IEEE Control Systems. 1991. pp. 52-56.
`Winey III. Calvin M.. “Computer Simulated Visual and
`Tactile Feedback as an Aid to Manipulator and Vehicle
`Contro ". MII‘. 1981.
`Iwata. Hiroo. “Pen-based Hapitc Virtual Environment.”
`Institute of Engineering Mechanics. Univerisity of Tsukuba.
`Japan. pp. 287-292.
`Schrnult. Brian et aL. “Application Areas for a Force-Feed-
`back Joystick.” ASME 1993. DSC-vol. 49. pp. 47-54.
`McAiIee. Douglas A..
`‘Teleoperator Systemlfelerobot
`Demonstrator: Force Reflecting Hand Controller Equipment
`Manual." JPL Jan. 1988. pp. 3-8. 11. and A—34.
`Howe. Robert D.. “Task Performance with a Dextrous
`Teleoperated Hand System." Proceedings of SPIE. Nov.
`1992. vol. 1833. pp. 1-9.
`Snow. E. et al.. “Compact Force-Reflecting Hand Control-
`ler.” .lPL.Apr. 1991.vol. 15.No. 3.ItemNo. l53.pp. 1-15a.
`Tan. Hong Z et aL. “Manual Resolution of Compliance
`When Work and Force Cues are
`ASME 1993.
`DSC-vol. 49. pp. 99-104.
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0003
`
`

`
`5,734,373
`Page 4
`
`Kotoku. Tetsuo. “A Predictive Display with Force Feedback
`and its Application to Remote Manipulation System with
`Transmission Time Display." IEEE 1992. Jul. 7-10. 1992.
`pp. 239-246.
`
`Russo. Massimo Andrea. ‘The Design and Implementation
`of a Three Degree-—of—Freedom Force Output Joystick.”
`
`Department of Mechanical Engineering. May 11. 1990. pp.
`9-40 & 96 & 97.
`Jones. L.A.. et 211.. “A Perceptual Analysis of Stiffness."
`Experimental Brain Research 1990. pp. 151-156.
`Has ser. Christopher John. ‘Tactile Feedback for a Force—Re—
`fleeting Haptic Display." The School of Engineering. Uni-
`versity of Dayton. Dec. 1995. pp. ii.i—xii & 1-96.
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0004
`
`

`
`U.S. Patent
`
`Mar. 31, 1993
`
`Sheet 1 of 21
`
`AUDIO OUT-
`
`PUT DEVICE
`
`FORCE FEEDBACK
`INTERFACE DEVICE
`— — — _ — _ _ _ _ . _ _ _ _ _ _ _ _ _ — _ _.._/-.......—I
`
`26
`
`LOCAL
`MICRO-
`
`PROCESSOR
`
`39
`OTHER
`INPUT
`
`38
`
`41
`
`'
`OBJECT
`
`_I
`
`I_
`
`__
`
`7
`
`|‘:fi'é"’q"":I\‘(’:'I‘E
`
`ACTUATORS
`
`IIIIlIIIIIIIIII
`
`T. __
`AN"T_|
`5 ULATEJ
`
`__:_L____
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0005
`
`

`
`U.S. Patent
`
`In3uM
`
`8m1...
`
`Sheet 2 of 21
`
`5,734,373
`
`MICRO-
`PROCESSOR
`
`OUTPUT
`
`ACTUATOR
`
`57
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0006
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 3 of 21
`
`5,734,373
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0007
`
`

`
`U.S. Patent
`
`Mar. 31, 1993
`
`Sheet 4 of 21
`
`5,734,373
`
`% 72
`
`74
`
`/0
`
`HOST COMPUTER
`
`76
`
`PROCESS HOST
`APPLICATION
`
`powER_Up HOST
`COMPUTER AND
`INTERFACE DEVICE
`
`MICROPROCESSOR
`
`86
`
`ACTIVATE
`INTERFACE DEVICE
`
`RECEIVE
`SENSOR DATA
`FROM MICRO-
`PROCESSOR
`
`CHANGE
`IN FORCE
`ON OBJECT RE-
`QUIRED?
`
`DETERMINE LOW-LEVEL
`FORCE COMMAND IN
`ACCORDANCE WITH A
`SELECTED REFLEX
`PROCESS, SENSOR DATA,
`HOST APPLICATION, AND
`OTHER DATA
`
`SEND LOW—LEVEL
`FORCE COMMAND TO
`MICROPROCESSOR
`
`READ RAW
`DATA FROM
`SENSORS
`
`PROCESS
`RAW DATA
`INTO SENSOR
`DATA
`
`SEND
`SENSOR
`DATA TO
`
`RECEIVE
`
`OUTPUT
`LOw_LEVEL
`FORCE COMMAND
`TO ACTUATORS
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0008
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 5 of 21
`
`5,734,373
`
`HOST COMPUTER
`
`I05
`
`PROCESS HOST
`APPLICATION
`
`pOwER.Up H031‘
`COMPUTER AND
`INTERFACE DEVICE
`
`MICROPROCESSOR
`
`715
`
`ACTIVATE
`INTERFACE DEVICE
`
`READ SENSOR
`DATA OF USER
`OBJECT FROM
`MICRO-
`PROCESSOR
`
`CHANGE
`IN TYPE OF FORCE
`ON OBJECT RE-
`QUIRED?
`
`DETERMINE
`HIGH LEVEL
`COMMAND
`
`ISSUE HIGH LEVEL
`FORCE COMMAND TO
`MICROPROCESSOR
`
`READ
`RAW
`DATA FROM
`SENSORS
`
`PROCESS RAW
`DATA INTO
`SENSOR DATA
`
`STORE AND
`SEND SENSOR
`DATA TO HOST
`
`RECEIVE
`NEW HIGH
`LEVEL FORCE
`COMMAND?
`
`SELECT REFLEX
`PROCESS BASED
`ON HIGH LEVEL
`FORCE
`COMMAND
`
`DETERMINE FORCE
`COMMAND TO
`ACTUATORS IN
`ACCORDANCE WITH
`REFLEX PROCESS.
`SENSOR/OTHER DATA,
`AND HIGH LEVEL
`FORCE COMMAND
`
`OUTPUT LOW-LEVEL
`FORCE COMMAND
`
`128
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0009
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 6 of 21
`
`5,734,373
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0010
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 7 of 21
`
`5,734,373
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0011
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`5,734,373
`
`1890 186a
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0012
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 9 of 21
`
`5,734,373
`
`ZO_._._QZOO
`
`ZO_._._n_ZOO
`
`zoEn_zoo
`
`zoEozoo
`
`><.Em>o
`
`><._mm>o
`
`><._mm>o
`
`\><._mm_>o
`
`ZO_._._n_ZOO
`
`><._mm_>o
`
`Emk.
`
`wank
`
`mm<._om=m
`
`mmm»m_2<m<..oz<oz<s_s_oo".0u._._s_<xm
`
`
`
`
`
`moz<s_s_oo1.0.3.200m:.<_.._
`
`
`
`Ez<mn_<mo.m:t.m
`
`
`
`
`
`.zoEE:n_.._.zm_o_n_n_m_oo._oz_mn_m-m
`
`
`
`8oz<m_o<m_o_m._tm-\l8m
`
`
`m/r_zo:<m:o.mQEzw<_>_.memo".m
`
`
`
`m_._>._.m.zo_.Em:o.»zm__o_u_"m_oo.o:._m
`
`
`
`
`
`
`
`fidfim.zo_»<m:o.mQEzw<_am._m<»mz:
`
`
`
`
`
`
`
`.zo_5mm_o.zo_.Em:o.ma:.:zo<§m»<mm_>:>
`
`
`
`
`
`
`
`.m._>._.m.>ozm:omm”_
`
`Azo_5mm_o
`
`
`
`
`
`_zoEE:o.mo3_zw<s:mOEOu_|>
`
`
`
`Am._>._.w.ZO_.r<m_DD_m_o:tz.,u<§m._mmo>>
`
`
`
`
`
`
`
`.zo:.omm_o.zoEE:o.mo::zo<s:womoufim
`
`
`
`
`
`
`
`m_m_"_oS<_2o._.5m
`
`
`
`Azo_.5mE_o.zo:.<m:o.mo2._zo<§.50:
`
`
`
`
`
`oz_moBmm
`
`ozfiokwmm
`
`oz_mn_m
`
`_._m__oo:._m
`
`xofiw
`
`m_._m<»mz:
`
`mo5m>
`
`zo_.Emm_>
`
`m._mmO>>
`
`zo_.5m
`
`._.._O_.
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0013
`
`

`
`Sheet 10 of 21
`
`5,734,373
`
`DISPLACEMENT
`
`_‘Figure 10a
`
`DISPLACEMENT
`
`figure 105
`
`DISPLACEMENT
`
`figure 10c
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0014
`
`

`
`U.S. Patent
`
`Mar. 31, 1993
`
`Sheet 11 of 21
`
`5,734,373
`
`FORCE
`
`320$
`
`DISPLACEMENT
`
`figure 11a
`
`DISPLACEMENT
`
`figure 115
`
`DISPLACEMENT
`
`figure 11::
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0015
`
`

`
`U.S. Patent
`
`Mar. 31, 1993
`
`Sheet 12 of 21
`
`5,734,373
`
`Y
`
`326 \\
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0016
`
`

`
`U.S. Patent
`
`la3mM
`
`m91
`
`Sheet 13 of 21
`
`5,734,373
`
`
`
`.moz<.cm_o-n_<zm.m._Cm.mn5:zo<s;m>oom0
`
`
`
`
`
`m....m\..
`
`Smmam
`
`mmmE_2<m<._oz<nz<s_s_oou_Om.Es_<xu
`
`
`
`
`
`maz<s_s_oo._oEzoozoEmo_.._
`
`
`
`fioz<5_o-n_<zw_m._>»m.mo_Ezo<§._.O>_n_
`
`
`
`
`
`
`
`m_oz<5_o-n_<zm.zoE..oo.__mo::zw<§mmEm<m:x
`
`
`
`
`
`
`
`fi._>._.m.>:mzm_a_mo::zo<s=mmakxmp
`
`
`
`
`
`
`
`fimzmm.z_o_mo.m_o_.Ezw<_2Vo._m__u_
`
`
`
`
`
`
`
`
`
`mmzmw,moz<_._n_s_oo.E_><mo.mm<s_.m:oo<n_
`
`
`
`
`
`
`
`
`
`mE_n_O.5<.zor5m.zo_5mm_a.ma::zo<s;z9.5m
`
`
`
`
`
`32%?
`
`moBm>
`
`m>oomo
`
`._.O>_D
`
`mm3._.Xm_._.
`
`mm_mm<m
`
`o._m_u_
`
`m._on_<n_
`
`zotsm
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0017
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 14 of 21
`
`5,734,373
`
`342‘
`
`S
`
`DISPLACEMENT
`
`figure 15
`
`—>l
`
`:<— B
`
`DISPLACEMENT
`
`THICKNESS
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0018
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 15 of 21
`
`5,734,373
`
`352
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0019
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`1|.2cm61amS
`
`5,734,373
`
`
`
`mm_m._.ms_<m<n_._Om._.ZOOv_o<mommu_women.
`
`
`
`
`
`
`
`mmmoomm._om._.zoo
`
`
`
`zoP<5n__2oom»<on_:
`
`mO._.<:._.o<oz<mmmoomm
`
`
`
`
`
`s__.Emo9<momoum:._.<hw
`
`oz_Eon_mm
`
`mmm.E_2<m<n_
`
`oz_.EoamE
`
`mmmoomm
`
`oz<_2s_oo
`
`wmmooma
`
`50:
`
`zoF<o_z:_2_2oo
`
`oz:om_9_o<m
`
`mwmoomm
`
`O2<momzmm
`
`<5502.2:
`
`oz_EonmE
`
`mmm:ms_<m<n_
`
`
`
`maz<_2s_oo0z_s_ooz_
`
`
`
`
`
`mmo»<:5<8%~‘Nk3$.n.§umm_».m.m_<T,_.oo
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0020
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 17 of 21
`
`5,734,373
`
`ESTABLISH
`COMMUNICATION
`LINK
`
`392
`
`SEND INTERFACE
`DEVICE INFORMATION
`TO HOST UPON
`REQUEST
`
`RECEIVE COMMANDS
`AND SET
`APPROPRIATE
`REPORTING
`PARAMETERS
`
`COMMAND
`RECEIVED?
`
`CONF|G_
`URA-I-ION
`COMMAND?
`
`SET APPROPRIATE
`REPORTING
`PARAMETERS
`
`SET FORCE
`PARAMETERS
`BASED ON FORCE
`COMMAND AND
`COMMAND
`PARAMETERS
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0021
`
`

`
`U.S. Patent
`
`Mar. 31, 1993
`
`Sheet 13 of 21
`
`5,734,373
`
`EXAMINE REPORTING
`AND FORCE
`PARAMETERS
`
`DETERMINE WHICH
`SENSORS TO BE
`READ FROM BOTH
`TYPES OF
`PARAMETERS
`
`ALWAYS
`COMPUTE VELOCITY
`AND/OR ACCEL-
`ERATION?
`
`COMPUTE
`VELOCITY ANDIOI-‘I
`ACCELERATION IF
`APPROPRIATE
`
`COMPUTE
`
`VELOCITY AND/OR
`ACCELERATION
`USING SENSOR
`READINGS AND
`TIME DATA
`
`STORE SENSOR
`DATA AND TIME
`DATA
`
`424
`
`WAIT IF NECESSARY
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0022
`
`

`
`U.S. Patent
`
`Mar. 31, 1993
`
`Sheet 19 of 21
`
`5,734,373
`
`432
`
`STREAM
`
`REQUEST
`FOR REPORT
`RECEIVED?
`
`TIME
`EXPIRED FOR
`REPORT?
`
`REPORT DATA
`ACCORDING TO
`REPORTING
`PARAMETERS AND
`REQUEST
`
`REPORT DATA
`ACCORDING TO
`REPORTING
`PARAMETERS
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0023
`
`

`
`U.S. Patent
`
`Mar. 31, 1998
`
`Sheet 20 of 21
`
`5,734,373
`
`@
`
`524
`
`TOTAL FORCE = 0
`
`CHECK FORCE
`
`FORCE PROCESS TO
`EXECUTE AND
`COMPUTE NEXT FORCE
`
`“3”
`
`468
`
`CONDITION3””
`
`VIOLATED?
`
`FORCE To
`ACTUATORS
`
`456
`
`SET ERROR FLAG
`
`ADD FORCE TO
`TOTAL FOFICE
`
`453
`
`ANOTHER
`once PROCESS
`TO EXECUTE FOR
`THIS AXIS?
`
`NO
`
`462
`
`STORE TOTAL FORCE
`FOR THIS AXIS
`
`ANOTHER
`AXIS TO COMPUT
`
`A TOT:/3g_RE?ORCE
`
`LIMIT TOTAL FORCE FOR
`EACH AXIS DEPENDING
`ON HARDWARE
`CONSTRAINTS AND
`ERROR FLAG
`
`SEND FORCES
`TO APPROPRIATE
`ACTU T R
`A O 8
`
`WAIT IF NECESSARY
`
`474
`
`figure 22
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0024
`
`

`
`U.S. Patent
`
`Mar. 31, 1993
`
`Sheet 21 of 21
`
`5,734,373
`
`
`
`mmmE_>_<m<n_memo“.mmm_.m_s_<m<n_
`ozfiam.oz_mo._.mmmmomom_._m__w_o:._m2
`
`
`
`
`
`
`
`cm»
`
`BEEHEEE.1
`
`
`
`:2:.Ex53_._m_wo:._m
`
`mmmw
`
`
`
` cm.x.8.:m_oo:._m\»momE»
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0025
`
`

`
`5,734,373
`
`1
`METHOD AND APPARATUS FOR
`CONTROLLING FORCE FEEDBACK
`INTERFACE SYSTEMS UTILIZING A HOST
`COMPUTER
`
`CROSS REFERENCE TO RELATED
`APPLICMTONS
`
`This application is a continuation-in-part of parent patent
`application Ser. No. 08/461,170. now U.S. Pat. No. 5.576.
`727. filed Jun. 5. 1995 on behalf of Louis B. Rosenberg et
`al.. entitled, “Three-Dimensional Mechanical Mouse."
`which is a continuation of application Ser. No. 08/092,974.
`filed Jul. 16. 1993. now abandoned; and co-pending parent
`patent application Ser. No. 08/534.791. filed Sep. 27. 1995.
`on behalf of Louis B. Rosenberg. entitled. “Method and
`Apparatus for Controlling Human-Computer Interface Sys-
`tems Providing Force Feedback.” both assigned to the
`assignee of this present application. and both of which are
`incorporated by reference herein.
`BACKGROUND OF THE INVENTION
`
`invention relates generally to interface
`The present
`devices between lunnans and computers. and more particu-
`larly to computer interface devices that provide force feed-
`back to the user.
`
`Computer systems are used extensively in many different
`industries to implement computer controlled simulations.
`games. and other application programs. More particularly.
`these types of games and simulations are very popular with
`the mass market of home consumers. A computer system
`typically displays a visual environment to a user on a display
`screen or other visual output device. Users can interact with
`the displayed environment to play a game. experience a
`simulation or “virtual reality" environment. or otherwise
`influence events or images depicted on the screen. Such user
`interaction can be implemented through the use of a human-
`computer interface device. such as a joystick. “joypad”
`button controller. mouse. trackball. stylus and tablet. or the
`like. that is connected to the computer system controlling the
`displayed environment. The computer updates the sirnula-
`tion or game in response to the user’s manipulation of an
`object such as a joystick handle or mouse. and provides
`feedback to the user utilizing the display screen and.
`typically. audio speakers.
`In some interface devices. tactile (“haptic”) feedback is
`also provided to the user. more generally known as “force
`feedback" These types of interface devices can provide
`physical sensations to the user manipulating the object of the
`interface device. Typically. motors or other actuators are
`coupled to the object and are connected to the controlling
`computer system. The computer system can provide forces
`on the object in conjunction with simulation/game events by
`sending control signals to the actuators. The computer
`system can thus convey physical sensations to the user in
`conjunction with other supplied feedback as the user is
`grasping or contacting the object of the interface device.
`Force feedback interface devices can thus provide a whole
`new modality for human-computer interaction.
`Force feedback inputloutput (I/O) devices of the prior art
`have concentrated on providing maximum haptic fidelity.
`i.e.. the realism of the tactile feedback was desired to be
`optimized. This is because most of the force feedback
`devices have been targeted at the specific needs of highly
`industrial applications. and not a mass consumer market. To
`attain such realism. mass market design concerns such as
`low size and weight.
`low complexity. programming
`
`2
`compatibility. low cost. and safety have been sacrificed in
`the prior art. As a result. typical force feedback interface
`devices include complex robotic mechanisms which require
`precision components and expensive actuators.
`An important concern for a force feedback interface
`device is communication bandwidth between the controlling
`computer and the interface device. To provide realistic force
`feedback. the complex devices of the prior art typically use
`high speed communication electronics that allow the con-
`trolling computer to quickly update force feedback signals to
`the interface device. The more quickly the controlling com-
`puter can send and receive signals to and from the interface
`device. the more accurately and realistically the desired
`forces can be applied on the interface object. In addition.
`using a high bandwidth communication interface. force
`feedback can be accurately coordinated with other supplied
`feedback. such as images on the video screen. and with user
`inputs such as movement of the object. activated buttons.
`etc. For example. a user can grasp and move a force
`feedbackjoystick in a simulation to control an image of a car
`to drive over a virtual bumpy surface displayed on a screen.
`The controlling computer should provide control signals to
`the actuators of the joystick quickly enough so that the
`surface feels as realistically bumpy as the designer of the
`simulation intended. If the control signals are too slow. a
`realistic feeling of bumpiness is more diflicult to provide.
`Also.
`the controlling computer needs a high bandwidth
`communication interface to accurately coordinate the sup-
`plied forces with the visual feedback on the screen. such as
`the moment on the screen when the car first contacts the
`bumpy surface. This high speed is likewise needed to
`accurately coordinate supplied forces with any input from
`the user. for example. to steer the car in particular directions.
`A problem is evident when prior art force feedback
`interface devices are provided to the mass consumer market.
`Most home computas have a built-in standard serial com-
`munication interfaces. such as an RS-232 or RS-422
`interface. that may conveniently be used to connect periph-
`erals like a force feedback interface device to the host
`computer. In addition. manufacturers prefer to provide
`peripheral devices that use these serial interfaces. since no
`additional hardware. such as interface cards. needs to be
`provided with such peripherals. The manufacturing cost of
`the peripheral device can thus be significantly reduced.
`However. these standard serial communication interfaces
`are typically quite slow (ie. have low bandwidth) compared
`to other commnnicaton interfaces. Realistic and accurate
`force feedback thus becomes difficult to provide by a con-
`trolling computer system to a prior art
`interface device
`conneaed through such a serial interface. For example. U.S.
`Pat. No. 5.184319. by J. Kramer. describes a force feedback
`device that applies forces to a user’s body parts. However.
`the Kramer device is typical of the prior art in that the host
`computer directly controls the actuators and directly
`receives the sensor data from the interface apparatus. Such
`a device is not suitable for a low bandwidth communication
`interface to achieve realistic force feedback
`
`Another problem with using prior art force feedback
`interface devices in the mass consumer market is the wide
`variety of computer platforms and processing speeds that are
`used on different computers and on the same computer at
`di.fl’erent times. The force sensations provided to a user by a
`force feedback interface device may feel different to a user
`on different computer platforms or microprocessors. since
`these different computers run at different speeds. For
`example.
`the force feedback controlled by a 100 MHz
`computer may be much different from the force feedback
`
`APPLE INC.
`EXHIBIT 1004 - PAGE 0026
`
`

`
`5,734,373
`
`3
`controlled by a 60 MHz computer due to the ditferent rates
`of processing control signals. even though these forces are
`intended to feel the same. In addition. the effective process-
`ing speed of one microprocessor can vary over time to
`provide inconsistent forces over multiple user sessions. For
`example. multitasking can vary or delay a microprocessors
`management of force feedback control signals depending on
`other programs that are running on the microprocessor.
`In addition. there is no standardized language or commu-
`nication protocol for communicating with force feedback
`devices. A software developer that wishes to provide force
`feedback to an interface in a software application must
`currently set up his or her own specialized commands and/or
`communications protocols and must implement the force
`feedback controlling instructions at a low level. This
`requires unnecessary time and expense in the development
`of software applications that
`include features directed
`toward force feedback interfaces.
`Therefore. a more realistic. cost etfective. and standard-
`ized alternative to force feedback interfaces and force feed-
`back control paradigms is desired for certain applications.
`SUMMARY OF THE INVENTION
`
`The present invention is directed to controlling and prc»
`viding force feedback to a user operating a human/computer
`interface device. The interface device is connected to a
`controlling host computer and includes a separate micro-
`processor local to the interface device. The local micropro-
`cessor receives high-level host commands and implements
`independent reflex processes.
`More particularly. a system and method of the present
`invention for controlling an interface apparatus manipulated
`by a user includes a host computer system for receiving an
`input control signal and for providing a host command. The
`host computer updates a host application process. such as a
`simulation or video game. in response to the input control
`signal. A microprocessor local to the interface apparatus and
`separate from the host computer receives the host command
`and provides a processor output control signal. An actuator
`receives the processor output control signal and provides a
`force along a degree of freedom to a user-manipulated
`object. such as a joystick. in accordance with the processor
`output control signal. A sensor detects motion of the user
`object along the degree of freedom and outputs the input
`control signal including information representative of the
`position and motion of the object. Preferably. the sensor
`outputs the input control signal to the local microprocessor.
`which outputs the input control signal to the host computer.
`The user object is preftnably grasped and moved by the user.
`and can include such articles as a joystick. mouse. simulated
`medical instrument. stylus. or other object. The user object
`can preferably be moved in one or more degrees of freedom
`using. for example. a gimbal or slotted yoke mechanism.
`where an actuator and sensor can be provided for each
`degree of freedom.
`The application process updated by the host computer
`system preferably includes application software that can be
`simulation software. game software. scientific software. etc.
`The host computer system displays images on a vi