(12) United States Patent
`Haanpaa et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,281,651 B1
`*Aug. 28, 2001
`
`US006281651B1
`
`(54) HAPTIC POINTING DEVICES
`
`(75)
`
`Inventors: Douglas Haanpaa, Ann Arbor; Gary
`Sieben, Dexter; Terry Cussenr Ann
`Arbor; Kirk Frfcrr Mike Drrrsrrrorcr
`borrr or Ypsrrarrrrr Charles J_ Jacobrrsr
`Ann Arbor» all 0f M1 (US)
`
`(73) A55ign°93 Immer5i011 C01‘P01‘3ti011> 5311 -7056, CA
`(US)
`
`(*) Notice:
`
`This patent issued on a continued pros-
`ecution application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`patent
`term provisions of 35 U.S.C.
`154("‘)(2)-
`
`0626634 A2
`4-34610/\
`W0 94/25157
`WO 95/20788
`wo 95/32459
`wo 96/22591
`
`............................... .. G06F/3/00
`11/1994 (EP)
`GOSG/9/02
`2/1992 (JP)
`A6113/5/00
`1/1994 (W0) --
`.. G06F/3/00
`8/1995 (W0) ..
`.. G06F/3/00
`11/1995 (wo) ..
`7/1996 (wo) .............................. G09G/5/00
`OTHER PUBLICATIONS
`
`
`
`M. Ouh—young, D. Beard, F. Brooks, Jr., “Force Display
`Performs Better than Visual Display in a Simple 6-D
`Docking Task,” IEEE, 1989, pp. 1462—6.
`W. Kim, P. Schenker, “A Teleoperation Training Simulator
`with Visual and Kinesthetic Force Virtual Reality,” Cal. Inst.
`Of Technology.
`
`(List continued on next page.)
`
`Subject to any disclaimer, the term of this
`Patent is extended or adjusted under 35
`U'S'C‘ 1540)) by 0 days’
`
`Primary Exmmner Karen Masrlr
`(74) Attorney, Agent, or Firm—Gitford, Krass, Groh,
`Sprinkle, Anderson & Citkovvski, PC
`
`(21) Appl. No.: 09/185,152
`
`(22)
`
`Filed;
`
`Nov_ 3, 1998
`
`(60)
`
`Related U.S. Application Data
`Provisional application No. 60/064,077, filed on Nov. 3,
`1997-
`Int. Cl.7 ......................................................... B25J 9/18
`(51)
`(52) U.S. Cl.
`.............................. .. 318/568.11; 318/568.16;
`xr
`(58) Field of Search ......................... 318/568.11, 568.16;
`414/7 5. 74/471. 434/45
`’
`’
`’
`
`(56)
`
`References Cited
`US. PATENT DOCUMENTS
`
`3,490,059
`3,795,150
`3,919,691
`
`73/133
`1/1970 Paulsen et al.
`. . . . . .. 74/5.4
`3/1974 Eckhardt
`. . . . . .
`
`11/1975 Noll
`................................ .. 340/172.5
`
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`
`(57)
`
`ABSTRACT
`
`A haptic pointing device includes a plurality of rigid, elon-
`gated proximal members, each connected to a separate rigid,
`elongated distal member through an articulating joint. The
`other end of each proximal member is coupled to an actuator
`such as a motor, causing that member to swing within a
`Separate plane PerP"‘r“l1.C“1*‘f trirnthe jhaffrt of the motor In
`response to a Comm Slgna’
`cn -0 cam 15 Intercon-
`
`1’
`‘
`.
`‘
`"
`gl
`mal members, the end-effector moves in space. In a pre-
`ferred embodiment, the device includes at least three proxi-
`mal members and three distal members, and the end-effector
`is coupled to a user—graspable element such as a stylus which
`retains a preferred orientation in space as the members are
`driven by the actuators. In a force-feedback application, the
`haptic pointing device further includes a position sensor
`associated with each degree of freedom, and haptic process-
`ing means interfaced to a virtual reality system or teleop-
`erations environment. Additional components may be pro-
`vided to increase flexibility, degrees of freedom, or both.
`
`0085518
`
`8/1983 (EP)
`
`.............................. .. GOSD/1/00
`
`25 Claims, 16 Drawing Sheets
`
`Visual Output
`__L_
`
`Haptic Input and
`Output (Pointing
`and Touching)
`
`
`
`Graphics Computer (which houses
`a Virtual World or CAD/CA_‘v\/I
`
`
`
`Network
`Connection
`
`
`
`Addition computers
`accessed over the network
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 1
`
`

`

`US 6,281,651 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`8/1980 Colston .......................... .. 340/365 L
`8/1983 La111 61 al
`434/45
`3/1984 101165 ~~~~ ~~
`188/378
`10/1984 Repperger ~
`244/223
`12/1985 Williams
`-~ 340/825
`8/1986 Joyce
`414/7
`12/1986 R6pp618616 211-
`244/230
`10/1988 K6Z610011i
`~~~~~ ~~
`414/735
`1/1989 Jan ......................................... .. 414/5
`1/1989 Cemenska et al.
`60/393
`3/1989 Hilton
`73/862.04
`MW” OWE - - - - - - -
`- - - ~MMW
`6/1989 Ichikawa et al.
`364/513
`8/1989 IWa1110to et al.
`.. 364/513
`8/1989 M6611611,11~ ~~~~~ ~~
`434/45
`9/1989 Affinito et al.
`.................... .. 340/710
`10/1990 MaCMinn ........................... .. 318/696
`10/1990 D6Mai0 6161-
`364/146
`1/1991 Kaye ....... ..
`244/243
`1/1991 L6l111161 -~
`318/685
`4/1991 l3111d6a
`~~~~~~~~ -- 414/6
`.... ..
`4/1991 Siva
`. 74/471 XY
`5/1991 Yoshinada at al,
`414/5
`9/1991 Behensky 6161-
`~- 434/45
`11/1991 R6pp61'861'
`~~~~~~~~ --
`244/175
`12/1991 Ferranti et al.
`.................... .. 244/228
`4/1992 McIntosh ...................... .. 318/568.22
`4/1992 Rose“ ------ --
`200/6 A
`4/1992 Cadoz 61 al ~ ~ ~ -
`~ ~ ~ ~~ 341/22
`5/1992 M01161i6f 61 al ~~
`273/448
`5/1992 Fung et al.
`. . . . . .
`. . . . . .. 414/5
`9/1992 Menahem
`. 74/471 XY
`9/1992 Burdea et al.
`......................... .. 414/5
`9/1992 Hollis, Jr. et al.
`................. .. 395/275
`10/1992 Cizewski ct al.
`..
`. 244/223
`1/1993 Ehrenfried .... ..
`432/52
`1/1993 Lang
`341/20
`2/1993 Kramer
`364/806
`318/432
`2/1993 G9°d 91 31-
`434/114
`2/1993 Rohen
`414/5
`3/1993 McAffee et a.
`316/561
`6/1993 Schuler
`......... ..
`6/1993 Radke et al.
`.................... .. 318/568.1
`7/1993 Chuang ............................ .. 74/471 X
`8/1993 Culver ..... ..
`74/471
`11/1993 Gregory et al.
`318/561
`2/1994 Fuller et al.
`.
`.. 434/45
`3/1994 Paley ............. ..
`345/163
`3/1994 Kramer et al.
`' 364/512
`................... .. 318/568.11
`8/1994 K '
`1
`1.
`.............................. .. 395/95
`8/1994 B:cI:Il1(1e:
`10/1994 Burdea el al.
`....... 414/5
`1/1995 Salcudean etal.
`318/568.11
`2/1995 Jacobus et al.
`318/568.11
`3/1995 Brimball
`....... ..
`.. 345/161
`3/1995 Taylor et al.
`606/130
`. 273/438
`4/1995 Katanics et al.
`5/1995 Schuler
`.................
`318/561
`9/1995 Massimino et al.
`..
`340/407.1
`10/1995 Jacobus et al.
`318/568.11
`12/1995 Lance et al.
`....................... .. 318/561
`2/1996
`Clark ct al.
`..
`341/20
`.. 345/163
`4/1996
`4/1996
`~~~~ -~ 345/156
`4/1996
`‘ 364/16701
`5/1996
`........ .. 414/5
`10/1996 Carmein ................................... 482/4
`11/1996 Rosenberg et al.
`................ .. 345/179
`11/1996 Jarvik . . . . . . . . . . . . . . . . .
`. . . . . .. 482/4
`12/1996 Renzi
`.............................. .. 340/407.1
`
`
`
`..
`
`
`
`..
`
`
`
`
`
`
`
`..
`
`
`
`4,216,467
`4,398,889
`4,436,188
`4,477,043
`4,560,983
`4,604,016
`4,632,341
`4,775,289
`4,795,296
`4,800,721
`4,811,608
`49184
`4,837,734
`4,853,874
`4,861,269
`4,868,549
`4,961,038
`4,962,448
`4,982,918
`4,983,901
`5,004,391
`5,007,300
`5,018,922
`5,044,956
`5,062,594
`5,076,517
`5,103,404
`5,107,080
`5,107,262
`5,116,051
`5,116,180
`5,142,931
`5,143,505
`5,146,566
`5,156,363
`5,180,351
`5,182,557
`5,184,319
`
`59185561
`511867629
`5,193,963
`5,220,260
`5,223,776
`5,228,356
`5,235,868
`5,264,768
`5,286,203
`5,296,871
`5,297,057
`5,339,799 *
`5,341,459
`5,354,162
`5,382,885
`5,389,865
`5,396,266
`5,397,323
`5,405,152
`5,414,337
`5,451,924
`5,459,382
`5,473,235
`5,491,477
`5,506,605
`5,512,919
`5,513,100
`5,516,249 *
`5,562,572
`5,576,727
`5,577,981
`5,583,478
`
`5,587,937
`5,589,828
`5,5g9;g54
`5,619,180
`5,623,582
`5,625,576
`5,629,594
`5,631,861
`5,642,469
`5,643,087
`5,666;138
`5,666’473
`5,701,140
`sjmgw
`5,714;978
`597215“)
`5,731,804
`597349373
`5,736;978
`5,739,811
`5,742;278
`5,767,839
`5,769,640
`5,781;172
`5,790,108
`5,805,140
`5,821,920
`558255308
`57828497
`5,844,392
`5,872,438
`5,889,670
`6,l09;407 =x<
`67lll;577 =x=
`
`OTHER PUBLICATIONS
`
`C. Winey, III, “Computer Simulated Visual and Tactile
`Feedback as an Aid to Manipulator and Vehicle Control”,
`Mass. Inst. Of Tech., 1981, pp. 1-79.
`J. Gotow, M. Friedman, M. Nagurka, J. Dolan, “Preception
`of Mechanical Properties at the Man—Machine Interface,”
`Carnegie Mellon Univ.’ 1987’ pp. 688_9_
`.
`..
`.
`.
`T K9‘°k“> K T“F“9> A F“J1kaWa> E1{"11°m9m M9d9h“g £91
`1116 l11t61a611V6 D1SPlaY(ElVllD)11S66l 111 T6l610b011C SYS16111s,
`1991, pp. 999-1004.
`S. Wiker, G. Vanderheiden, S. Lee, S. Arndt, “Development
`of Tactile Mice for Blind Access to Computers: Importance
`of Stimulation Locus, Object Size3, and Vibratactile Display
`Resolution”, Univ. Of Wisconsin, 1991, pp. 708-12.
`,,
`.
`.
`.
`M'.O“h_y°ung’ Force Dlsplay 1“ Molecular Dockmg ’
`U““’- Of_N-C-1 1999- “
`.
`_
`.
`Y,AdaCh1,N,Matsu1, _S611s01Y l3V‘<1l11at1011 0fV11111alllaP116
`Push—Butt0ns”, Suzuki Motor Corp.
`P. Fischer, R. Daniel, K. Siva, “Specification and Design of
`Input Devices for Teleoperation, ” IEEE, 1990, pp. 540-5.
`<4
`~
`L‘ Ffmsenberg’. Ylrltual fixtures aS.t°°1S Kt’ e,,nhSal§1I‘E ‘l’,p‘;’m°r
`P?” Ormance 1“
`‘’ epresence envlromen 5’
`6 ‘ma’
`H1P“14l°r T°9h“91°g)’> 1993
`.
`.
`M.Yamak1ta,H.Sh1basat0, K. Furuta, “Te1e—V1rtual Reality
`of Dynamic Mechanical Model,” IEEE, July 1992, pp.
`1103-10.
`
`H. Tan, N. Durlach, Y. Shao, M. Wei, “Manual Resolution of
`Compliance When Work and Force Cues are Minimized,”
`ASME> 19931 PP‘ 99404"
`“
`.
`F- 13100115, 11-, M- 01111-Y01111g, 1- B61161, P- 16101116, P101661
`GROPE-Haptic Displays
`for Scientific Visualization”,
`Computer Graphics, vol. 24, No. 4, Aug. 1990, pp. 177-85.
`
`APPLE INC.
`
`364/578
`12/1996 Massie et al.
`.. 341/20
`12/1996 Armstrong
`345/161
`12/1996 Teal
`................. N
`340/407.1
`4/1997 Massinlino et al.
`395/99
`4/1997 Rosenberg ...... ..
`364/578
`4/1997 Massie et al.
`..
`. 318/568.11
`5/1997 Jacobus etal.
`.
`364/406
`5/1997 Kramer
`........... ..
`.. 395/99
`6/1997 Ilannaford etal.
`7/1997 Marcus etal.
`....................... .. 463/38
`9/1997 Culver ................................ u 345/161
`9/1997 Wallace ........... N
`345/420
`12/1997 Rosenberg et al.
`345/156
`1nm8cmnam_m.
`m1%fl&
`2/1998 Yamanaka et al.
`345/157
`2/1998 Rosenberg et al.
`345/161
`3/1998 Rosenberg ......................... .. 345/156
`3/1998 Rosenberg et al_
`345/161
`4/1998 Hasser et al.
`345/173
`4/1998 Rosenberg etal.
`345/161
`4/1998 Chen et al.
`.
`345/l56
`6/1998 Rosenberg ..
`345/161
`6/1998 Jacobus etal.
`434/262
`7/1998 Engel et all
`‘
`345/164
`8/1998 Salcudean etal.
`345/184
`9/1998 Rosenberg etal.
`345/161
`10/1998 Rosenberg etal.
`345/156
`10/1998 Rosenberg ........................... ._ 341/20
`10/1998 Martin et all
`318/567
`12/1998 Peurach eta].
`. 318/568.17
`2/1999 Roston ...... ..
`. 318/568.11
`3/1999 Schuler et al.
`.... .. 364/186
`8/2000 Boekmann el all
`. 318/568.11
`8/2000 Zlles et al_
`glg/568
`
`
`
`.
`
`.
`
`
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 2
`
`

`

`US 6,281,651 B1
`Page 3
`
`B. Hannaford, Z. Szakaly, “Force-Feedback Cursor Con-
`trol,” NASA Tech Brief, vol. 13, No. 11, Nov. 1989.
`M. Ohu-Young, M. Minsky, M. Behensky, F. Brooks, Jr.
`“Creating an Illusion of Feel: Control Issues in Force
`Display,” Univ. of N. Carolina, Sept. 1989, pp. 1-14.
`S. Fisher, M. McGreevy, J. Humphries, W. Robinett, “Vir-
`tual Environment Display System”, NASA Ames Research
`Center, Oct. 1986.
`J. Herndon, D. Jelatis, C. Jennrich, H. Martin, P. Satterlee,
`Jr. “The State-of-the-Art Model M-2 Maintenance Sys-
`tem,” p. 59-66.
`M. Ouh-young, M. Pique, J. Hughes, N. Srinivasan, F.
`Brooks, Jr. “Using a Manipulator For Force Display in
`Molecular Docking”, Univ. of N. Carolina, IEEE, 1988, PP.
`1824-9.
`S. Su, R. Furuta, “The Virtual Panel Architecture: A 3 D
`Gesture Framework,” 1993, pp. 387-93.
`S. Jacobson, F. Smith, D. Backman, E. Iversen, “High
`Performance, High Dexterity, Force Reflective Teleoperator
`II,”Univ. of Utah, 1991.
`A. Bejczy, W. Kim, S. Venema, “The Phantom Robot:
`Predictive Displays for Teleoperation with Time Delay”,
`IEEE, 1990, pp. 546-50.
`H. Tan, M. Srinivasan, B. Eberman, B. Chen, “Human
`Factors for the Design of Force—Reflecting Haptic Inter-
`faces,” ASME, 1994.
`J. Colgate, P. Grafing, M. Stanley, G. Schenkel, “Implemen-
`tation of Stiff Virtual Walls in Force—Reflecting Interfaces,
`”Sept. 1993, Northwestern Univ.
`H. Iwata, H. Noma, “Volume Haptization,” IEEE, 1993, pp.
`16-18.
`
`L. Rosenberg, B. Adelstein, “Perceptual Decomposition of
`Virtual Haptic Surfaces,” Proc. IEEE Sym. on Res. Frontiers
`in Virtual Reality, Oct. 1993.
`L. Rosenberg, “The Use of Virtual Fixtures as Perpetual
`Overlays to Enhance Operator Performance in Remote Envi-
`ronments,” U.S. Air Force, Sept. 1992.
`L. Rosenberg, “Virtual haptic overlays enhance performance
`in telepresence tasks”, Sanford Univ., 1994.
`G. Burdea, J. Zhuang, “Dextrous telerobotics with force
`feedback —an overview,” Rutgers Univ., 1991.
`W. Kim, A. Bejczy, “Graphics Displays for Operator Aid in
`Telemanipulation,”IEEE, 1991, pp. 1059-67.
`G. Burdea E. Roskos, D. Gomez, N. Langrana, “Distributed
`Virtual Force Feedback, ” IEEE, May 1993, pp. 25-44.
`P. Kilpatrick, “The Use of a Kinesthetic Supplement in an
`Interactive Graphics System,” Univ. of N. Carolina, 1976.
`Immersion Corporation Brochure, “3D Human Interface
`Tool” .
`
`A. Kelly, S. Salcudean, “On the Development of a Force-
`Feedback Mouse and its Intergration into a Graphic User
`Interface,” Univ. of British Columbia, 1991.
`M. Minsky, M. Ohu—young, O. Steele, F. Brooks, Jr., M.
`Behensky, “Feeling and Seeing: Issues in Force Display”,
`pp. 235-70.
`L. Rosenberg, “Perceptual Design of a Virtual Rigid Surface
`Contact”, Stanford University, Apr. 1993, pp. 1-40.
`M. Akamatsu, S. Sata, “Multimodal Mouse: AMo11se-Type
`Dcvicc with Tactile and Force Display”, Presence, vol. 3,
`No. l, 1994, pp. 73-80.
`B. Adelstein, M. Rosen, “Design and Implementation of a
`Force Reflecting Manipulandum for Manual Control
`Research”, NASA-Ames Research Center, 1992.
`
`H. Iwata, “Artificial Reality with Force-feedback: Devel-
`opment of Desktop Virtual Space with Compact Master
`Manipulator”, Computer Graphics, vol. 24, No. 4, Aug.
`1990, pp. 165-170.
`L. Rosenberg, S. Brave, “The Use of Force Feedback to
`Enhance Graphical User Interfaces”, Stereoscopic Displays
`and Virtual Reality Systems III, Proc. SPIE, 1996, pp.
`243-248.
`
`A. Kelly, S. Salcudean, “Magic Mouse: Tactile and Kines-
`thetic Feedback in the Human-Computer Interface using an
`Electromagnetically Actuated Input/Output Device”, U. of
`Br. Columbia, 1993, pp. 1-26.
`B. Schmult, R. Jebens, “Application Areas for a Force-
`Feedback Joystick”, ASME Advances in Robotics, Mecha-
`tronics, and Ilaptic Interfaces, vol. 49, 1993, pp. 47-54.
`M. Russo, “The Design and Implementation of a Three
`Degree-of-Freedom Force Output Joystick”, May, 1990.
`J. Payctte, V. Hayward, C. Ramstcin, D. Bcrgeron, “Evalu-
`ation of a Force Feedback (Haptic) Computer Pointing
`Device in Zero Gravity”, ASME, 1996, pp. 547-53.
`K. Hirota, M. Hirose, “Development of Surface Display”,
`IEEE, 1993, pp. 256-62.
`S. Salcudean, P. Drexel, D. Ben-Dov, A. Taylor, P.
`Lawrence, “a Six Degree-of-freedom, Hydraulic, One Per-
`son Motion Simulator”, University of British Columbia.
`D. Tavkhelidze, N. Davitashvili, “Kinematic Analysis of
`Five-Link Spherical Mechanisms”, Mechanism and
`Machine Theory, V0l. 9, 1974, pp. 181-90.
`L. Rosenberg, S. Brave, “The Use of Force Feedback to
`Enhance Graphical User Interfaces”.
`M. Bostrom, S. Singh, C. Wiley, “Design ofAn Interactive
`Lumbar Puncture Simulator With Tactile Feedback”, IEEE,
`1993, pp. 280-86.
`T. Howe, D. Kontarinis, “Task Performance with a Dextrous
`Teleoperated Hand Systen”, Proceedings of SPIE presented
`at Telemanipalator Technology ’92, vol. 1833, Nov. 1992,
`pp. 1-9.
`B. Adelstein, M. Rosen, “A High Performance Two
`Degree-of-Freedom Kinesthetic Interface”, Massachusetts
`Institute of Technology, 1992, pp. 108-12.
`H. Iwata, “Pen-based Haptic Virtual Environment”, IEEE,
`1993, pp. 287-92.
`V. Hayward, J . Choksi, G. Lanvin, C. Ramstein, “Design and
`Multi-Objective Optimization of a Linkage for a Haptic
`Interface”, Mc Gill University, pp. 359-68.
`P. Millman, J. Colgate, “Design of a Four Degree-of-Free-
`dom Force—Reflecting Manipulandum with a Specified
`Forcc/Torquc Workspace ”, IEEE Proceedings of Interna-
`tional Conference on Robotics and Automation, 1991, pp.
`1488-92.
`
`J . Battcr, F. Brooks, Jr., “Gropc-1: A Computer Display to
`the Sense of Feel”, Proc. IFIP Congress, 1971, pp. 759-63.
`P. Buttolo, B. Hannaford, “Pen-Based Force Display for
`Precision Manipulation in Virtual Environments”, IEEE,
`1995, pp. 217-224.
`L. Rosenberg, “A Force Feedback Programming Primer”,
`1997, pp. 1-176.
`R. Ellis, O. Ismaeil, M. Lipsett, “Design and Evaluation of
`a High-Performance Prototype Planar Haptic Interface”,
`Advances in Robotics, Mechatronics, and IIaptic Interfaces,
`vol. 49, Dec. 1993, pp. 55-64.
`C. Ramstein, V. Hayward, “The Pantograph: ALarge Work-
`space Haptic Device for a Multi111odal Human-Computer
`Interaction”, Computer-Human Interaction, 1994.
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 3
`
`

`

`US 6,281,651 B1
`Page 4
`
`L. Rosenberg, “The Use of Virtual Fixtures to Enhance
`Operator Performance in Time Delayed Teleoperation”, U.S.
`Air Force, Mar. 1993.
`W. Atkinson, K. Bond, G. Tribble, III, K. Wilson, “Com-
`puting with Feeling”, Comput, & Graphics, Vol. 2, 1977, pp.
`97-103.
`
`B. Hannaford, L. Wood, D. McAfee, H. Zak, “Performance
`Evaluation of a Six Axis Generalized Force—Reflecting
`Teleoperator”, IEEE Transctions on Systems, Man and
`Cybernetics, Vol. 21, No. 3, May—Jun. 1991, pp. 620-33.
`
`* cited by examiner
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 4
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 1 of 16
`
`US 6,281,651 B1
`
`
`
`Visual Output
`
`____L_
`
`
`
`Haptic Input and
`Output (Pointing
`and Touching)
`
`
`Network
`Connection
`
`
`
`Addition computers
`accessed over the network
`
`Figure 1
`
`APPLE INC.
`
`
`
`Graphics Computer (which houses
`a Virtual World or CAD/CAM
`
`Model)
`
`
`
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 5
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 2 of 16
`
`US 6,281,651 B1
`
`/‘
`
`Figure 2
`(Prior Art)
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 6
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 3 of 16
`
`US 6,281,651 B1
`
`
`
`'
`
`Top platform moves in X, Y, Z,
`pitch, roll, yaw based on
`' ' ' ' ' ' ' translation in d1 through d6
`
`_ _ _ _ . - - - ' '
`
`Ball or
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 7
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 4 of 16
`
`US 6,281,651 B1
`
`
`
`X, Y, Z, pitch, roll, yaw
`
`5|
`
`Ball or Universal Joints
`
`% Q4
`
`3 '
`
`Rotary Actuators
`rqu (To
`ers or Motors)
`
`Figure 4
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 8
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 5 of 16
`
`US 6,281,651 B1
`
`
`
`
`
`Ball or Universal Joints
`
`Rotary Actuators
`(Torquers or Motors)
`
`
`
`
`
`Wires feed through a hollow
`universal joint (implemented
`with a coil spring)
`
`Stylus or grip (with buttons
`and deacl—man sensors)
`
`Figure 6
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 9
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 6 of 16
`
`US 6,281,651 B1
`
`Photo
`Emitter/Detector Pair
`
`As s emb ly
`
`Metal Strip (with
`Etched Strip
`Pattern on It)
`
`
`
`Drum (attached to
`Motor Shaft)
`
`Linkage Arm (also
`attached to Motor
`
`Shaft or Drum)
`
`Figure 7
`
`Buttons
`
`
`
`/
`
`Home Stud
`
`
`
`Dead—Man Sensor
`
`(one on each side)
`
`Figure 8
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 10
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 7 of 16
`
`US 6,281,651 B1
`
`922
`
`924
`
`Status lndicators
`(LEDS)
`
`Interface
`(Serial)
`
`I 1
`
`Haptics Processor
`(DSP)
`
`Deadman Sensors
`(IR Transceivers) _
`
`User Controls
`(Buttons)
`
`916
`
`902
`
`906
`
`908
`
`
`
`V5 em
`Performance
`Sensors
`(Temperature
`‘ Sensor)
`
`I
`C t
`‘F
`Astaqaligr Dtiglgcfis
`(Closed ITOOD
`Current Drivers)
`
`Displacement
`
`Decoders
`(Quadrature
`Counters)
`
`912
`
`914
`
`910
`
`I’
`
`
`
`
`
`Rotational
`
`Actuators
`(Momrs)
`
`Sensors
`(Encoders)
`
`Figure 9
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 11
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 8 of 16
`
`US 6,281,651 B1
`
`
`
`Application
`
`Application Programmer
`Interface Layer
`
`Device Driver
`
`Layer
`
`Communications
`Layer
`
`Device Interface
`
`Layer
`
`
`
`
`1010
`
`1012
`
`1020
`
`
` 1022
`
`Figure 10
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 12
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 9 of 16
`
`US 6,281,651 B1
`
`
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 13
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 10 of 16
`
`US 6,281,651 B1
`
`50
`
`Figure 12
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 14
`
`
`
`

`

`U.S. Patent
`
`Aug. 28, 2001
`
`em
`
`1cl01
`
`US 6,281,651 B1
`
`Sm
`
`1:1
`
`1mt1|.
`
`61|.
`
`Figure 13
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 15
`
`
`
`

`

`
`
`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 12 of 16
`
`US 6,281,651 B1
`
`Horizontal (x,y) Plane
`
`
`
`Figure 14A
`
`Figure 14B
`
`Vertical (y,z) Plane
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 16
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 13 of 16
`
`US 6,281,651 B1
`
`Figure 15
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 17
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 14 of 16
`
`US 6,281,651 B1
`
`Q.
`
`Figure 16
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 18
`
`
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 15 of 16
`
`US 6,281,651 B1
`
`
`
`
`
`Figure 17
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 19
`
`

`

`U.S. Patent
`
`Aug. 28,2001
`
`Sheet 16 of 16
`
`US 6,281,651 B1
`
`APPLE INC.
`
`A L
`
`2
`
`Xi’ Y1’ Z1
`
`X73 Y? Z’:
`
`Figure 18
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 20
`
`

`

`1
`HAPTIC POINTING DEVICES
`
`2
`SUMMARY OF THE INVENTION
`
`US 6,281,651 B1
`
`REFERENCE TO RELATED APPLICATION
`
`This application claims priority of U.S. provisional appli-
`cation Serial No. 60/064,077, filed Nov. 3, 1997, the entire
`contents of which are incorporated herein by reference
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to force feedback
`and, in particular, to a family of haptic interface devices
`which exploit multiple redundant links to position a scribe or
`stylus in space.
`
`BACKGROUND OF THE INVENTION
`
`Force feedback devices originated in various specialized
`ways in the 1960s with teleoperations. Most of these devices
`were “replica” devices, wherein a smaller controlling or
`master robot was moved by an operator to control movement
`of a larger slaved robot. Forces detected at the slave were
`then also fed back to the operator through the master robot’s
`actuators. Such prior art is substantially described in U.S.
`Pat. No. 5,389,865 to Jacobus et al.
`In the late 1980s, NASA funded several experiments
`using force feedback devices which were not configured as
`having identical versions of a slave device. This was impor-
`tant at the time because an astronaut may want to control a
`number of kinds of space-based robots and cranes from one
`“universal” controller. To make this universal controller
`
`concept work, the master controller was connected logically
`to the slave through a network of computers which were
`capable of translating the master kinematics typically into
`Cartesian coordinates and from Cartesian to slave kinemat-
`ics and back the other way.
`Once this computer controller is in place on the master
`side of the system, it becomes possible to send inputs from
`the master (joystick, wheel, yoke, etc.) to a simulated slave
`rather
`than a real one, and to accept
`forces from the
`simulation for application to the master as well. This is one
`innovation described in U.S. Pat. Nos. 5,389,865 and 5,459,
`382 to Jacobus et al. As disclosed by Jacobus, this simulation
`need not be a real device, like a crane or robot, but may be
`a simulated automobile, boat, plane, or weapon. It can even
`be a simulation of a person performing tasks in a virtual
`world, such as walking, handling things, touching surfaces.
`U.S. Pat. Nos. 5,459,382 and 5,389,865 describe an early
`device and method for providing users with a touch or tactile
`interface into a virtual world which allows the user to touch
`
`virtual objects, or objects which are not real, but rather are
`described by a model which resides inside a computer
`system. U.S. Pat. No. 5,389,865 and 5,629,594, and U.S.
`application Ser. No. 08/845,375 elaborate on these devices
`and the software architecture responsible for expressing
`abstract virtual models in the computer as forces created
`based on the position of the devices. U.S. application Ser.
`No. 08/861,080 describes in detail how a software architec-
`ture represents abstract virtual objects in terms of superpo-
`sitions of virtual geometrical entities and dynamic special
`“feel” effects. U.S. application Ser. No. 08/859,877
`describes how virtual objects and geometrical entities are
`built using CAD/CAM and geometrical design methods.
`U.S. application Ser. No. 08/859,157 describes how touch or
`haptic attributes are parameterized and represented using
`graphical user interface elements. All of the patents and
`applications discussed above are set forth herein in their
`entirety by reference.
`
`Broadly, this invention resides in techniques and systems
`for combining tactile feedback elements to create a family of
`haptic or touch user interface devices. In particular, these
`combinations exploit multiple redundant links to position a
`scribe or stylus in a single, unique multi-degree of freedom
`position and orientation while presenting a force or force
`and torque to the stylus. The stylus, in turn, is pressed on the
`CAD/CAM or designer’s hand, simulating tactile interaction
`with a virtual object within the device’s volume of move-
`ment.
`
`In a basic configuration, a haptic pointing device accord-
`ing to the invention includes a plurality of rigid, elongated
`proximal members, each connected to a separate rigid,
`elongated distal member through an articulating joint. The
`other ends of each proximal member is coupled to an
`actuator causing that member to swing within a separate
`plane in response to a control signal. An end-effector is
`interconnected to the second end of each distal member
`
`through an articulating joint, such that as the actuators move
`the proximal members, the end-effector moves in space. The
`various articulating joints may be universal
`joints, ball
`joints, of composed of a flexible material.
`In a preferred embodiment,
`the device includes three
`proximal members and three distal members, enabling the
`end effector to move in three-dimensional space. The actua-
`tors preferably take the form of motors having a shaft
`perpendicular to the plane of movement of the proximal
`member coupled thereto. Different style motors, including
`pancake motors, may be used. In one configuration,
`the
`end-effector is coupled to a user-graspable element having a
`preferred orientation in space, with the arrangement of the
`members and joints being such that the element retains the
`preferred orientation as the end effector is moved by the
`actuators.
`
`In a force-feedb ack application, the haptic pointing device
`further includes a position sensor associated with each
`proximal member, and haptic processing means operative to
`output an electronic signal indicative of the position of the
`end-effector in space, and receive an electronic signal caus-
`ing the actuators to move the end—effector to a point in space.
`The haptic processing means may in turn be interfaced to a
`virtual reality system including descriptions of virtual
`objects, enabling a user to interact with the objects through
`the end effector. Alternatively, the haptic processing means
`may be interfaced to a slave system including a separate set
`of actuators and encoders, enabling a user to interact with
`the objects through the end effector as part of a teleopera-
`tions environment.
`
`In alternative embodiments, additional components may
`be provided to increase flexibility, degrees of freedom, or
`both. For instance, the end-effector may be further coupled
`to a user-graspable element such as a handle or stylus to
`provide one or more additional degrees of freedom associ-
`ated with roll, pitch and yaw. The system may include a pair
`of base platforms, each with its own set of actuated proximal
`and distal members for example, six sets of proximal and
`distal members may be utilized, three sets associated with
`each of the two base platforms. In all cases, the system may
`further conveniently include a docking station to receive the
`end effector for calibration purposes.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a drawing used to show how the invention may
`be employed to support the touching of geometric objects
`described within a virtual world interfaced to a haptic or
`tactile device;
`
`10
`
`15
`
`20
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`APPLE INC.
`
`APPLE INC.
`EXHIBIT 1014 - PAGE 21
`
`

`

`US 6,281,651 B1
`
`3
`FIG. 2 is a drawing of a prior-art stylus which was found
`to be impractical for various reasons;
`FIG. 3 is a drawing used to introduce and illustrate a
`“Stewart platform,” forming a basis for inventive principles
`disclosed herein;
`FIG. 4 is one modification of the Stewart platform which
`uses direct-drive rotary actuators as part of a six-degree-of-
`freedom design;
`FIG. 5 is a different modification of the Stewart platform
`utilizing three actuators and linkages as part of a three-
`degree-of-freedom device according to the invention;
`FIG. 6 illustrates how, through the use of hollow tubes and
`flexible guides, wires may be dressed to an actuator/
`manipulator;
`FIG. 7 is a drawing as seen from an oblique perspective
`illustrating how semiconductor optical sensors and detectors
`may be mounted on a metallic strip for encoding purposes
`according to the invention;
`FIG. 8 illustrates a preferred stylus incorporating actuator
`buttons and a “dead man” switch;
`
`FIG. 9 is a block diagram illustrating major functional
`electronic subsystems according to the invention;
`FIG. 10 is a simplified representation of the way in which
`application and device layers are integrated according to a
`software aspect of the invention;
`FIG. 11 shows a physical embodiment of a three-degree-
`of—freedom linkage;
`FIG. 12 shows an aspect of the invention wherein, in
`particular, a rotary actuator is coupled to a rigid rod or tube;
`FIG. 13 is a perspective View of a completed three-
`degree-of-freedom apparatus;
`FIGS. 14A and 14B are diagrams which help to illustrate
`device kinematics in solving from Cartesian coordinates to
`link vector locations;
`
`FIG. 15 shows the outline diagram of an alternative
`embodiment of a three-axis design implemented with DC
`pancake motors;
`FIG. 16 depicts the design of a three-axis measurement-
`only end elfector which may be substituted for the simple
`stylus of previous figures; and
`FIG. 17 shows a variation in modifying a conventional
`Stewart platform by connecting two three-degree-of-
`freedom systems together through a common linkage and
`stylus.
`FIG. 18 shows link kinematics for system configuration
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`FIG. 1 suggests how the invention may be employed to
`support touching of geometric objects described within a
`virtual world representation by a geometric probe logically
`attached to a haptic or tactile device. The representation to
`support touch is synchronized with a comparable represen-
`tation of the world for visualization or graphical rendering,
`enabling the world to be both seen and touched. Because of
`the analogy to visualization entailed by this method,
`the
`generation of forces from the act of touching virtual objects
`is called haptic rendering (as a analog to visual rendering),
`and the act of both touching and viewing the objects (or
`other data) is referred to as browsing, and the software
`package which does the browsing is called a browser.
`
`10
`
`15
`
`20
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`Generally speaking, the basis for the invention resides in
`alternative haptic device designs having at least some of the
`following properties:
`low inertia and weight to provide very high haptic fidelity;
`a desktop-oriented force feedback with range of motion
`comparable to a conventional computer mouse (the 2D
`workspace was to be nominally equal
`to that of a
`mousepad);
`compatibility with 3D CAD data (i.e. CAD/CAM soft-
`ware representations);
`low cost; and
`simple and reliable operation.
`One promising haptic design is shown in FIG. 2. This design
`approach, while used in existing commercial devices,
`proved to be heavy, had high inertia, and exhibited signifi-
`cant backlash in the drive train when a low cost actuation
`
`motor/gear system was used. The design also exhibits prob-
`lems with structure flex and timing belt/cable tensioning.
`Although these problems could be corrected with effort, the
`overall design is too far from ideal overall.
`The preferred concept began as a miniature conventional
`“Stewart platform,” as shown in FIG. 3, except that we
`employed rotary actuators, instead of linear actuators. FIG.
`4 depicts one platform design embodiment modified to use
`direct-drive rotary actuators as part of a six-degree-of-
`freedom design. FIG. 5 depicts an alternative embodiment
`using only three actuators and linkages for a three-degree-
`of—freedom device.
`
`From the three-degree-of-freedom design concept of FIG.
`5 the device kinematics were computed. This was done
`through a genetic algorithm which varied motor size,
`location, and link arm segment lengths to produce accept-
`able forces levels in all directions at all points in preferred
`the work envelope (nominally a volume with a mouse pad
`sized base area and height). The genetic algorithm returned
`5 configurations which were evaluated as “the best,” con-
`sidering the kinematics and evaluation cost criteria.
`One of these configurations was selected as optimal based
`on inspection and layout considerations and, most
`importantly, the clearance of obstacles by the links. One
`problem with the mathematical analysis was that some
`positions wit