US007027032B2
`
`United States Patent
`US 7,027,032 B2
`(10) Patent No.:
`(12)
`Rosenbergetal.
`(45) Date of Patent:
`*Apr. 11, 2006
`
`
`(54) DESIGNING FORCE SENSATIONS FOR
`FORCE FEEDBACK COMPUTER
`APPLICATIONS
`
`3,220,121 A
`
`11/1965 Cutler
`
`(Continued)
`
`EP
`
`FOREIGN PATENT DOCUMENTS
`0626634 A2
`11/1994
`
`(75)
`
`(73) Assignee:
`
`(*) Notice:
`
`claimer.
`
`Inventors: Louis B. Rosenberg, Pleasanton, CA
`(US); Dean C. Chang, Palo Alto, CA
`(US)
`(Continued)
`Immersion Corporation, San Jose, CA
`OTHER PUBLICATIONS
`(US)
`Cursor Walso,“Designer’s Corner-Useful Technology for
`Subjectto any disclaimer, the term ofthis
`on
`;
`patent is extended or adjusted under 35
`Your Idea File,” Design News, Mar. 7, 1993, p. 63.
`US.C. 154(b) by 0 days.
`(Continued)
`as patent is subject to a terminal dis-
`Primary Examiner—Alexander Eisen
`.
`(74) Attorney, Agent, or Firm—Thelen Reid & Priest LLP;
`(21) Appl. No.: 10/782,878
`David B. Ritchie
`(22)
`Filed:
`Feb. 23, 2004
`ABSTRACT
`6)
`(65)
`Prior Publication Data
`US 2004/0160415 Al=Aug. 19, 2004
`Related U.S. Application Data
`Adesign interface tool for designing force sensations for use
`;
`;
`.
`with a host computer and force feedback interface device. A
`(63) Continuation of application No. 09/734,630, filed on
`force feedback device is connected to a host computerthat
`Dec. 11, 2000, now Pat. No. 6,697,086, which is a
`displays the interface tool. Input from a user is received in
`continuation-in-part of application No. 08/566,282,
`the interface to select a type of force sensation to be
`filed on Dec. 1, 1995, now Pat. No. 5,734,373, and a
`commanded by a host computer and output by a force
`continuation-in-part of application No. 08/846,011,
`feedback interface device. Input, such as parameters, is then
`filed on Apr. 25, 1997, now Pat. No. 6,147,674.
`received from the user which designs and defines physical
`(51)
`Int. Cl.
`characteristics of the selected force sensation. A graphical
`(2006.01)
`G09G 5/00
`representation of the characterized force sensation is dis-
`(52) US. CM. cereessesseseenee
`345/156; 345/157; 715/701
`Playedon the hostcomputerwhich provides a visual dem-
`(58) Field of Classification Search .............. 345/156,
`fethe of a Tee!
`of
`the f ee ne easa so
`345/157, 161; 715/700-866, 701, 702; 436/36-38
`at
`the user can view al ellect
`OF parameters On sac
`tore
`See applicationfile for complete search history.
`sensation. The characterized force sensation is output to a
`user manipulatable object of the force feedback device so
`References Cited
`that the user can feel the designed force sensation, where the
`graphical representation is updated in conjunction with the
`output ofthe force sensation. The user can iteratively modify
`force sensation characteristics and feel the results, and store
`the characterized force sensations.
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`2,906,179 A
`2,972,140 A
`3,157,853 A
`
`9/1959 Bower
`2/1961 Hirsch
`11/1964 Hirsch
`
`36 Claims, 25 Drawing Sheets
`
`Windows Help
`
`PERE EES
`@ Conditions ae
`eo Damper
`Oretion
`oO Inertia
`
`
`
`Smart Pop
`Vector Force
`@ Dynamics
`Level [35.09—]Time [6,084170];{0.436467
`
`1
`
`APPLE 1023
`
`Trigger 408 Repeat interval
`
`1
`
`APPLE 1023
`
`

`

`PPPPrPrrrErrPEPEErErrEPPeeeEEEEeeeeeEeEEEPeeeereeeeeeeeeemeremememeee
`
`5,309,140PPPPrPrrEEEPEPEPEEPSerEEeeEEEEEEeeeEPPSeeeeeeeeeeeeeeemerPrP
`
`4,945,501
`4,961,138
`4,961,267
`4,962,591
`4,982,504
`4,983,786
`5,007,085
`5,007,300
`5,019,761
`5,022,384
`5,022,407
`5,035,242
`5,038,089
`5,040,306
`5,044,956
`5,050,608
`5,072,361
`5,076,517
`5,078,152
`5,088,046
`5,088,055
`5,095,303
`5,103,404
`5,107,080
`5,116,051
`5,116,180
`5,126,948
`5,128,671
`5,131,844
`5,132,672
`5,139,261
`5,142,506
`5,142,931
`5,143,505
`5,146,566
`5,148,377
`5,165,897
`5,175,459
`5,178,012
`5,181,181
`5,182,557
`5,184,306
`5,184,319
`5,185,561
`5,186,629
`5,186,695
`5,187,874
`5,189,806
`5,193,963
`5,204,824
`5,209,131
`5,212,473
`5,220,260
`5,223,776
`5,228,356
`5,230,623
`5,235,868
`5,240,417
`5,243,266
`5,251,127
`5,251,156
`5,259,120
`5,259,894
`5,262,777
`5,264,768
`5,271,290
`5,275,174
`5,275,565
`5,283,970
`5,289,273
`5,296,846
`5,299,810
`
`7/1990
`10/1990
`10/1990
`10/1990
`1/1991
`1/1991
`4/1991
`4/1991
`5/1991
`6/1991
`6/1991
`TN991
`8/1991
`8/1991
`9/1991
`9/1991
`12/1991
`12/1991
`1/1992
`2/1992
`2/1992
`3/1992
`4/1992
`4/1992
`5/1992
`5/1992
`6/1992
`TN992
`TN992
`TN992
`8/1992
`8/1992
`9/1992
`9/1992
`9/1992
`9/1992
`11/1992
`12/1992
`1/1993
`1/1993
`1/1993
`2/1993
`2/1993
`2/1993
`2/1993
`2/1993
`2/1993
`3/1993
`3/1993
`4/1993
`5/1993
`5/1993
`6/1993
`6/1993
`TN993
`TN993
`8/1993
`8/1993
`9/1993
`10/1993
`10/1993
`11/1993
`11/1993
`11/1993
`11/1993
`12/1993
`1/1994
`1/1994
`2/1994
`2/1994
`3/1994
`4/1994
`5/1994
`
`
`
`Bell et al.
`Gorniak
`Herzog
`Zeller et al.
`Soderberg etal.
`Stevenset al.
`Greaniaset al.
`Siva
`Kraft
`Freels
`Horch et al.
`Franklin et al.
`Szakaly
`McMurtry et al.
`Behensky et al.
`Watanabeetal.
`Davis et al.
`Ferranti et al.
`Bondetal.
`McMurtry
`Oyama
`Clark etal.
`McIntosh
`Rosen
`Moncrief etal.
`Funget al.
`Mitchell et al.
`Thomas, Jr.
`Marinaccioet al.
`Clark
`Openiano
`Edwards
`Menahem
`Burdeaet al.
`Hollis, Jr. et al.
`McDonald
`Johnson
`Danial et al.
`Culp
`Glynn
`Lang
`Erdman etal.
`Kramer
`Goodet al.
`Rohen
`Mangseth et al.
`Takahashietal.
`McMurtry et al.
`McAffee etal.
`Fujimaki
`Baxter
`Louis
`Schuler
`Radkeetal.
`Chuang
`Guthrie et al.
`Culver
`Smithson et al.
`Kasagamiet al.
`Raab
`Heieretal.
`Chapman et al.
`Sampson
`Lowetal.
`Gregory et al.
`Fischer
`Cook
`Moncrief
`Aigner
`Lang
`Ledley
`Pierce et al.
`Everett, Jr. et al.
`
`US 7,027,032 B2
`
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`1/1970
`2/1970
`6/1970
`10/1970
`11/1971
`4/1975
`6/1975
`9/1975
`9/1975
`10/1975
`11/1975
`3/1976
`11/1978
`11/1978
`4/1979
`TN9T9
`8/1980
`12/1980
`4/1981
`6/1982
`8/1983
`5/1984
`8/1984
`10/1984
`10/1984
`11/1984
`4/1985
`10/1985
`11/1985
`2/1986
`4/1986
`6/1986
`7/1986
`7/1986
`12/1986
`1/1987
`3/1987
`3/1987
`6/1987
`TN987
`8/1987
`10/1987
`11/1987
`11/1987
`12/1987
`6/1988
`9/1988
`11/1988
`12/1988
`12/1988
`1/1989
`1/1989
`2/1989
`3/1989
`4/1989
`6/1989
`TN989
`9/1989
`11/1989
`12/1989
`12/1989
`1/1990
`1/1990
`1/1990
`3/1990
`3/1990
`6/1990
`6/1990
`6/1990
`7/1990
`7/1990
`
`Paulsen et al.
`Hirsch
`Corlyon et al.
`Stevensen
`Kagan
`Crockeret al.
`Fister et al.
`Hightower
`Diamondet al.
`Feder
`Noll
`Eaton
`Jones
`Lowthorp
`Burson
`Salisbury, Jr.
`Colston
`Hall et al.
`Schwellenbach
`Barnes
`Lam etal.
`Hayashi
`Foerst
`Repperger
`Davies
`Vavra
`Acklam et al.
`Mabusth
`Fraignieret al.
`Fraser et al.
`Boothroyd
`Davies
`Hladky etal.
`Watson
`Reppergeret al.
`Sheldon et al.
`Iwano
`Herrington et al.
`Slocum
`Koontz
`Lindbom
`Moriyasu
`Grahn et al.
`de Vries et al.
`Alban
`Zanetti
`Trieb etal.
`Olson
`Brunnett
`Selinko
`Miessler etal.
`Cemenskaetal.
`Kendig etal.
`Hilton
`Bell et al.
`LaBicheet al.
`Blood.
`Affinito et al.
`Duimel
`Embach
`Enderle et al.
`McIntosh
`Tomelleri
`Culver
`Meenen,Jr.
`Hon
`Baker
`McIntosh
`Kley
`Sapia
`Blood.
`
`3,490,059
`3,497,668
`3,517,446
`3,531,868
`3,623,064
`3,875,488
`3,890,958
`3,902,687
`3,903,614
`3,911,416
`3,919,691
`3,944,798
`4,125,800
`4,127,752
`4,148,014
`4,160,508
`4,216,467
`4,236,325
`4,262,549
`4,333,070
`4,398,889
`4,448,083
`4,464,117
`4,477,043
`4,477,973
`4,484,191
`4,513,235
`4,550,221
`4,550,617
`4,571,834
`4,581,491
`4,593,470
`4,599,070
`4,601,206
`4,632,341
`4,638,798
`4,653,011
`4,654,648
`4,676,002
`4,679,331
`4,688,983
`4,703,443
`4,704,909
`4,708,656
`4,713,007
`4,750,487
`4,769,763
`4,787,051
`4,791,934
`4,794,392
`4,798,919
`4,800,721
`4,803,413
`4,811,608
`4,819,195
`4,839,838
`4,849,692
`4,868,549
`4,879,556
`4,885,565
`4,888,877
`4,891,764
`4,891,889
`4,896,554
`4,907,970
`4,907,973
`4,930,770
`4,934,694
`4,935,728
`4,942,545
`4,945,305
`
`2
`
`

`

`US 7,027,032 B2
`
`Page 3
`
`5,334,027 A
`8/1994 Wherlock
`5,341,459 A
`8/1994 Backes
`5,354,162 A
`10/1994 Burdeaet al.
`5,379,663 A
`1/1995 Hara
`5,384,460 A
`1/1995 Tseng
`5,389,865 A
`2/1995 Jacobuset al.
`5,396,266 A
`3/1995 Brimhall
`5,396,267 A
`3/1995 Bouton
`5,397,323 A
`3/1995 Tayloret al.
`5,402,582 A
`4/1995 Raab
`5,405,152 A
`4/1995. Katanicsetal.
`5,412,880 A
`5/1995 Raab
`5,414,337 A
`5/1995 Schuler
`5,417,696 A
`5/1995 Kashubaet al.
`5,428,748 A
`6/1995 Davidson et al.
`5,429,140 A
`7/1995 Burdeaet al.
`5,436,542 A
`7/1995 Petelin et al.
`5,436,622 A
`7/1995 Gutman etal.
`5,436,640 A
`7/1995 Reeves
`5,437,607 A
`8/1995 Taylor
`5,445,166 A
`8/1995 Taylor
`5,451,924 A
`9/1995 Massiminoetal.
`5,459,382 A
`10/1995 Jacobusetal.
`5,466,213 A
`11/1995 Hogan etal.
`5,467,763 A
`11/1995 McMahonet al.
`5,482,051 A
`1/1996 Reddyet al.
`5,512,919 A
`4/1996 Araki
`5,513,100 A
`4/1996 Parkeretal.
`5.526.480 A
`6/1996 Gibsonetal.
`5.547.382 A
`8/1996 Yamasakietal.
`5,550,562 A
`8/1996 Aoki et al.
`5,551,701 A
`9/1996 Bouton et al.
`5,570,111 A
`10/1996 Barrett etal.
`5,575,761 A
`11/1996 Hajianpour
`5,576,727 A
`11/1996 Rosenberg et al.
`28oot A 7loos eon
`5,596,347 A
`1/1997 Robertson et al.
`5,623,582 A
`4/1997 Rosenberg
`5.625.576 A
`4/1997 Massie etal.
`5,629,594 A
`5/1997 Jacobuset al.
`5,642,469 A
`6/1997 Hannafordet al.
`5,643,087 A
`7/1997 Marcuset al.
`5,666,138 A
`9/1997 Culver
`5,666,473 A
`9/1997 Wallace
`5,690,582 A
`11/1997 Ulrich et al.
`eet ‘
`Mvloo, Cosenbea et al.
`5.7 14.978 A
`9/1998 Yamanaka et al.
`5,721,566 A
`2/1998 Rosenberg et al.
`5,734,373 A
`3/1998 Rosenberg etal.
`5,739,811 A
`4/1998 Rosenbergetal.
`5,742,278 A
`4/1998 Chenet al.
`5,754,023 A
`5/1998 Rosten et al.
`5,755,577 A
`5/1998 Gillio
`2ee ‘
`.hoon Satinelll al
`5,767,839 A
`6/1998 Rosenberg
`5,769,640 A
`6/1998 Jacobusetal.
`5,781,172 A
`7/1998 Engelet al.
`5,785,630 A
`7/1998 _Bobick etal.
`5,790,108 A
`8/1998 Salcudean et al.
`5,802,353 A
`9/1998 Avila et al.
`5,805,140 A
`9/1998 Rosenberg et al.
`5,808,601 A
`9/1998 Leah etal.
`5,825,308 A
`10/1998 Rosenberg
`eataos A
`Ishoo facobus a
`5.889.670 A
`3/1999 Schuleret al.
`5,959,613 A
`9/1999 Rosenberg,et al.
`6,111,577 A
`8/2000 Zilles et al.
`6,160,489 A
`12/2000 Perry etal.
`6,169,540 B
`1/2001 Rosenberg etal.
`
`6,285,351 Bl
`6,292,170 Bl
`6,422,941 Bl
`6,433,771 BI*
`
`9/2001 Chang et al.
`9/2001 Chang et al.
`97/2002 Thorneret al.
`8/2002 Yocum etal... 345/156
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`GB
`JP
`JP
`JP
`JP
`JP
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`
`0875819 Al
`2254911
`H12-185278
`4-3461
`H4-8381
`H5-192449
`H7-24147
`0 349 086 Al
`WO 9502801
`WO 9520787
`WO 9520788
`WO 9532459
`WO 9616397
`WO 9622591
`WO 9642078
`WO 9731333
`
`11/1998
`10/1992
`7/1990
`1/1992
`1/1992
`8/1993
`1/1995
`1/1990
`1/1995
`8/1995
`8/1995
`11/1995
`5/1996
`7/1996
`12/1996
`8/1997
`
`OTHER PUBLICATIONS
`“Foot-Operated Mouse,” IBM Technical Disclosure Bulle-
`tin, Apr. 1986, vol. 28, No. 11.
`ae,
`.
`.
`Im-
`High Performance Model of the Immersion Probe,”
`mersion Probe-MD.TM., Immersion Corporation (1994).
`“Proceedings of the IFIP Congress 65,” International
`Federation for Information Processing, Information Process-
`ing 1965, vol. 3, New York, May. 24-29, 1965, p. 506.
`“The Personal Digitizer.TM.,”Immersion Human Interface
`Corporation 1994.
`“Useful Technology for Your Idea File,” Design News, Mar.
`7, 1994, p. 63.
`.
`3D Human Interface Tool, Immersion Probe.TM., Immer-
`sion Human Interface Corporation 1994.
`Adachi, Yoshitaka et al., “Sensory Evaluation of Virtual
`Haptic Push-Buttons,” Technical Research Center, Suzuki
`Motor Corporation, Nov. 1994.
`Adelstein Bernard D. et al., “A High Performance Two
`Degree-of-Freedom Kinesthetic Interface,” Massachusetts
`Institute of Technology 1992, pp. 108-112.
`Akamatsu et al., “Multimodal Mouse: A Mouse-Type
`Device with Tactile & Force Display,” Presence, vol. 3, No.
`1, 1994, pp. 73-80. a
`Atkinson, W.et al., “Computing with Feeling,” Comp. &
`Graphics, vol. 2, 1976, pp. 97-103.
`Batter, James J. et al., “Grope-1: A Computer Display to the
`Sense fo Feel,” pp. TA-4-188-TA-4-192.
`Bejezy et al., “The Phantom Robot: Predictive Displays for
`Teleoperation with Time Display,” CA Institute of Technol-
`ogy, IEEE 1990, pp. 546-550,
`Brooks Jr, Frederick et al.,
`“Project GROPE-Haptic
`Displays for Scientific Visualization,” Computer Graphics,
`vol. 24, No. 4, 1990, pp. 177-185.
`Burdea, Grigore et al., “A Portable Dextrous Master with
`Force Feedback,” Presence: Teleoperators and Virtual
`Environments, MIT Press, Jun. 1991.
`Burdea, Grigore et al., “Dextrous Telerobotics with Force
`Feedback-An Overview,” Robotica 1991, vol. 9.
`Buttolo, Pietro et al., “Pen-Based Force Display for Preci-
`sion Manipulation in Virtual Environments,’ IEEE Mar.
`1995, pp. 1-8.
`Colgate, J. Edward et al., “Implementation of Stiff Virtual
`Walls
`in Force-Reflecting
`Interfaces,’ Northwestern
`University, 1993.
`
`3
`
`

`

`US 7,027,032 B2
`
`Page 4
`
`Ellis, R.E. et al., “Design and Evaluation of a High-
`Performance Prototype Planar Haptic Interface,” ASME
`Dec. 3, 1993, DSC-vol. 49, pp. 55-64.
`Fischer, Patrick et al., “Specification and Design of Input
`Devices for Teleoperation,” 1990.
`Fisher, S.S. et al., “Virtual Environment Display System,”
`ACM Interactive 3D Graphics, Oct. 1986.
`Gotow, J.K., et al., “Preception of Mechanical Propertiesat
`the Man-Machine Interface,” IEEE 1987, pp. 688-689.
`Hannafordet al., “Force Feedback Cursor Control,” NASA
`Tech Brief, vol. 13, No. 11, Item #21, 1989, pp. 1-4.
`Hannaford, Blake et al., “Performance Evaluation of a
`Six-Axis Generalized Force-Reflecting Teleoperator,” IEEE
`May/Jun. 1991, vol. 21, No. 3, pp. 620-633.
`Hasser, Christopher John, “Tactile Feedback for a Force-
`Reflecting Haptic Display,’ The School of Engineering,
`University of Dayton, Dec. 1995, pp. 1ii-xii & 1-96.
`Herndon, J.N. et al., “The State-of-the-Art Model M-2
`Maintenance System,” Proceedings of the 1984 National
`Topical Meeting on Robotics and Remote Handling in
`Hostile Environments, American Nuclear Society, pp. 59-
`65.
`
`Hiroo Iwata, “Artificial Reality with Force-Feedback:
`Development of Desktop Virtual Space with Compact
`Master Manipulator,’ Computer Graphics, vol. 24, No. 4,
`1990, pp. 165-170.
`Hirota etal., “Development Of Surface Display,” University
`Tokyo, IEEE, 1993, pp. 256-262.
`Howe, Robert D., “Task Performance with a Dextrous
`Teleoperated Hand System,” Proceedings of SPIE, Nov.
`1992, vol. 1833, pp. 1-9.
`Iwata, Hiroo et al, Volume Haptization, IEEE 1993, pp.
`16-18.
`
`Kelley et al., “On the Development of a Force-Feedback
`Mouse & It’s Integration into a Graphical User Interface,”
`Int’l1 Mechanical Engineering Congress and Exhibition,
`1994, pp. 1-8.
`Kelley et al., Magic Mouse: Tactile & Kinesthetic Feedback
`in the Human-ComputerInterface using an Electromagneti-
`cally Acutuated Input/Output Device, 1993, University Brit-
`ish Columbia.
`Kilpatrick et al., “ The Use of Kinesthetic Supplementin an
`Interactive Graphics System,” University of North Carolina,
`1976, pp. 1-174.
`Kim, WonS. et al., “Graphics Displays for Operator Aid in
`Telemanipulation,” IEEE 1991, pp. 1059-1067.
`Kim, WonS. et al., A Teleoperation Training Simulator with
`Visual and Kinesthetic Force Virtual Reality.
`Kotoku, Tetsuo et al., “Environment Modeling for the
`Interactive Display (EMID) Used in Telerobotic Systems,”
`IEEE Nov.3-5, 1991, pp. 99-1004.
`Kotoku, Tetsuo, “A Predictive Display with Force Feedback
`and its Application to Remote Manipulation System with
`Transmission Time Display,” IEEE 1992, Jul. 7-10, 1992,
`pp. 239-246.
`Krueger, Myron W., Artificial Reality 1988, pp. 54-75.
`Meyer, Kennethetal., “A Survey of Position Trackers,” The
`Massachusetts Institute of Technology 1992, Presence, vol.
`1, No. 2.
`Minsky, Margaretet al., “Feeling & Seeing: Issues in Force
`Display,” ACM 1990, pp. 235-242, 270.
`Munch et al., “Intelligent Control for Haptic Displays,”
`Eurographics ’96, Blackwell Publishers, vol. 15, No. 3,
`1996, pp. C-217-226.
`
`Noll, A. Michael, “Man-Machine Tactile Communication
`Dissertation,” Polytechnic Institute of Brooklyn, Jun. 1971,
`pp. 1-88.
`Ouh-Youngetal., “Using A Manipulator for Force Display
`in Molecular Docking,”, University North Carolina, IEEE
`1988, pp. 1824-1829.
`Ouh-young, Ming et al. “Creating an Illusion of Feel:
`Control Issues in Force Display,” University of N. Carolina,
`1989, pp. 1-14.
`Ouh-young, Ming et al., “Force Display Performs Better
`than Visual Display in a Simple 6-D Docking Task,” IEEE
`1989, pp. 1462-1466.
`Ouh-young, Ming et al., “Using a Manipulator for Force
`Display in Molecular Docking,” IEEE 1988, pp. 1824-1829.
`Ouh-Young, Ming,“Force Display in Molecular Docking,”
`Chapel Hill 1990, pp. 1-85.
`Payette et al., “Evaluation Of Force Feedback Computer
`Pointing device in Zero Gravity,” DSC-vol. 58, Proc. of
`ASMEDynamics Systems and Control Division, 1996, pp.
`547-553.
`
`Repperger, D.W., “Active Force Reflection Devices in
`Teleoperation”, IEEE Control Systems, 1991, pp. 52-56.
`Rosenberg et al., “Commercially Viable Force Feedback
`Controller for Individuals with Neuromotor Disabilities,”
`Crew Systems Directorate, AL/CF-TR-1997-0016, 1996,
`pp. 1-33.
`Rosenberg et al., “Perceptual Decomposition of Virtual
`Haptic Surfaces,” Proc.
`IEEE Symposium on Research
`Frontiers in Virtual Reality, 1993.
`Rosenberg et al., “The use of force feedback to enhance
`graphicaluser interfaces,” Stereoscopic Displays and Virtual
`Reality Systems, Proc. SPIE, 1996, pp. 243-248.
`Rosenberg, “Virtual haptic Overlays enhance performance
`in telepresence tasks,” Stanford University, 1994.
`Rosenberg, L., “Perceptual Design of a Virtual Rigid
`Surface Contact,” Air Force Material Command, AL/CF-
`TR-1995-0029, 1993, pp. 1-39.
`Rosenberg, L., “The Use of Fixtures to Enhance Operator
`Performance in Time Delayed Teleoperation,” Air Force
`Material Command, AL/CF-TR-1994-0139, 1993, pp. 1-45.
`Rosenberg, Louis B., “The Use of Virtual Fixtures as
`Perceptual Overlays to Enhance Operator Performance in
`Remote Environments,” Air Force Material Command, Sep.
`1992, pp. 1-42.
`Rosenberg, Louis B., “Virtual Fixtures as Tools to Enhance
`Operator Performance in Telepresence Environments,” SPIE
`Telemanipulator Technology, 1993.
`Rosenberg, Louis B., “Virtual Haptic Overlays Enhance
`Performance in Telepresence Tasks,” SPIE 1994.
`Rosenberg, Louis B., Crew Systems Directorate Biodynam-
`ics and Biocommunications Division Wright-Patterson, Air
`Force Material Command, Mar. 1993, pp. 1-45.
`Schmult, B. et al., “Application Areas for a Force-Feedback
`Joystick,” DSC-vol. 49, Advances in Robotics Mechatron-
`ics, and Interfaces, ASME 1993, pp. 47-54.
`Smith, Geoffrey, “Call
`It Palpable Progress,” Business
`Week, Oct. 9, 1995, pp. 93, 96.
`Snow,E. et al., “Compact Force-Reflecting Hand Controller,
`” JPL, Apr. 1991, vol. 15, No. 3, Item No. 153, pp. 1-15a.
`Su, S. Augustine et al., “The Virtual Panel Architecture: A
`3D Gesture Framework,” IEEE 1993, pp. 387-393.
`Tan, Hong et al., “Human Factors for the Design of Force-
`Reflecting Haptic Interfaces,” MIT, 1994.
`
`4
`
`

`

`US 7,027,032 B2
`
`Page 5
`
`Tavkhelidze, D.S., “Kinematic Analysis of Five-Link
`Spherical Mechanisms,” Mechanism and Machine Theory,
`1974, vol. 9, pp. 181-190.
`Wilker, Steven F. et al., “Development of Tactile Mice for
`Blind Access to Computers: Importance of Stimulation of
`Locus, Object Size, and Vibrotactile Display Resolution,”
`Proceedings of the Human Factors Society 35th Annual
`Meeting 1991, pp. 708-712.
`Winey III et al., “Computer Simulated Visual & Tactile
`Feedback as an aid to Manipulator & Vehicle Control,” MIT,
`1981, pp. 1-79.
`Patrick, “Design, Construction, and Testing of a Fingertip
`Tactile Display for Interaction with Virtual and Remote
`Environments,” Master ofScience Thesis, MIT, Aug. 1990,
`archived Nov. 8, 1990.
`Calder, “Design of A Force-Feedback Touch-Introducing
`Actuator For Teleoperator Robot Control,” Bachelor of
`Science Thesis, MIT, May 1983, archived Jun. 23, 1983.
`Wiker, “Teletouch Display Development: Phase 1 Report,”
`Technical Report 1230, Naval Ocean Systems Center, San
`Diego, Jul. 1988.
`Bliss, “Optical-to-Tactile Image Conversion for the Blind,”
`IEEE Transactions on Man-Machine Systems, vol. MMS-
`11, No. 1, Mar. 1970.
`Johnson, “Shape-Memory Alloy Tactile Feedback Actuator,
`” Armstrong Aerospace Medical Research Laboratory,
`AAMRL-TR-90-039, Aug., 1990.
`Kontarinis et al., “Tactile Display of Vibratory Information
`in Teleoperation and Virtual Environments,” PRESENCE,
`4(4):387-402, Harvard Univ., 1995.
`Aukstakalnis etal., “Silicon Mirage: The Art and Science of
`Virtual Reality,” ISBN 0-938151-82-7, pp. 129-180, 1992.
`Eberhardt et al., “Inducing Dynamic Haptic Perception by
`The Hand: System Description and Some Results,” DSC-
`vol. 55-1, Dynamic Systems and Control: vol. 1, ASME
`1994.
`
`Gobelet al., “Tactile Feedback Applied to Computer Mice,”
`International Journal ofHuman-Computer Interaction, vol.
`7, No. 1, pp. 1-24, 1995.
`Pimentel et al., “Virtual Reality: through the new looking
`glass,” 2,4, Edition: McGraw-Hill, ISBN 0-07-050167-X,
`pp. 41-202, 1994.
`“Cyberman Technical Specification,” Logitech Cyberman
`SWIFT Supplement to Logitech Mouse Technical Reference
`and Programming Guide, Apr. 5, 1994.
`Ouhyounget al., “The Development of A Low-Cost Force
`Feedback Joystick and Its Use in the Virtual Reality
`Environment,” Proceedings ofthe Third Pacific Conference
`on Computer Graphics and Applications, Pacific Graphics
`°95, Seoul, Korea, Aug. 21-24, 1995.
`Kaczmarek et al., “Tactile Displays,” Virtual Environment
`Technologies, Chap. 9, pp. 349-414.
`Lake, “Cyberman from Logitech,”at http://www.1biblio.org/
`GameBytes/Issue2 1/greviews/cyberman.html, 1994.
`“Component Maintenance Manual With Illustrated Parts
`List, Coaxial Control] Shaker Part No. C-25502,” Safe Flight
`Instrument Corporation, Revised Jan. 28, 2002 (3 pages).
`“Technical Manual Overhaul
`Instructions With Parts
`Breakdown, Coaxial Control Shaker Part No. C-25502,”
`Safe Flight Instrument Corporation, Revised Jul. 15, 1980
`(23 pages).
`Scannell, “Taking a Joystick Ride,” Computer Currents,
`Boston Edition, vol. 9, No. 11, Nov. 1994.
`
`Rosenberg, “Virtual Fixtures: Perceptual Overlays Enhance
`Operator Performance in Telepresence Tasks,” Ph.D Dis-
`sertation, Stanford University, Jun. 1994.
`Balgrie, “Electric Control Loading—A low Cost, High
`Performance Alternative,’ Proceedings of Interservice/
`Industry Training Systems Conference, pp. 247-254, Nov.
`6-8, 1990.
`Iwata, “Pen-based Haptic Virtual Environment,” 0-7803-
`1363-1/93 IEEE, pp. 287-292, 1993.
`Russo, “The Design and Implementation of a Three Degree
`of Freedom Force Output Joystick,” MITLibraries Archives
`pp. 1-131, May 1990, archived Aug. 14, 1990.
`Brooksetal., “Hand Controllers for Teleoperation—AState-
`of-the-Art Technology Survey and Evaluation,’
`JPL
`Publication 85-11, NASA-CR-175890; N85-28559, pp. 1-
`84, Mar. 1, 1985.
`Jones et al., “A perceptual analysis of stiffness,” ISSN
`0014-4819
`Springer
`International
`(Springer-Verlag);
`Experimental Brain Research, vol. 79, No. 1, pp. 150-156,
`1990.
`
`Burdeaetal., “Distributed Virtual Force Feedback, Lecture
`Notes for Workshop on Force Display in Virtual Environ-
`ments and its Application to Robotic Teleoperation,” 1993
`IEEE International Conference on Robotics and Automa-
`tion, pp. 25-44, May 2, 1993.
`Snow et al., Model-X Force-Reflecting-Hand-Controller,
`NT Control No. NPO-17851; JPL Case No. 7348, pp. 1-4
`with 45 pages of attachments, Jun. 15, 1989.
`Ouh-Young,
`“Force Display in Molecular Docking,”
`Doctoral Dissertation, University of North Carolina at
`Chapel Hill, UMI Order No. 9034744, p. 1-369, 1990.
`Tadros, “Control System Design for a Three Degree of
`Freedom Virtual Environment Simulator Using Motor/
`Brake Pair Actuators,” MIT Archive, pp. 1-88, Feb. 1990,
`archived Aug. 13, 1990.
`Caldwell et al., “Enhanced Tactile Feedback (Tele-Taction)
`Using a Multi-Functional Sensory System,” 1050-4729/93,
`pp. 955-960, 1993.
`Adelstein et al. “Design and Implementation of a Force
`Reflecting Manipulandum of Manual Control research,”
`DSC-vol. 42, Advances in Robotics, pp. 1-12, 1992.
`Gotow et al., “Controlled Impedance Test Apparatus for
`Studying Human Interpretation of Kinesthetic Feedback,”
`WAI1-11:00, pp. 332-337.
`Stanleyet al., “Computer Simulation of Interacting Dynamic
`Mechanical Systems Using Distributed Memory Parallel
`Processors,” DSC-vol. 42, Advances in Robotics, pp. 55-61,
`ASME 1992.
`
`Russo, “Controlling Dissipative Magnetic Particle Brakes in
`Force Reflective Devices,” DSC-vol. 42 Advances in Robot-
`ics, pp. 63-70, ASME 1992.
`Kontarinis et al., “Display of High-Frequency Tacile
`Information to Teleoperators,” 7elemanipulator Technology
`and Space Telerobotics, Won S. Kim, Editor, Proc. SPIE vol.
`2057, pp. 40-50, Sep. 7-9, 1993.
`Patrick et al., “Design and Testing of A Non-reactive,
`Fingertip, Tactile Display for Interaction with Remote
`Environments,” Cooperative Intelligent Robotics in Space,
`Rui J. deFigueiredo et al, Editor, Proc. SPIE vol. 1387, pp.
`215-222, 1990.
`Adelstein, “A Virtual Enivironment System For The Study
`of Human Arm Tremor,” Ph.D. Dissertation, Dept. of
`Mechanical Engineering, MIT, Jun. 1989, archived Mar. 13,
`1990.
`
`5
`
`

`

`US 7,027,032 B2
`Page 6
`
`Bejezy, “Sensors, Controls, and Man-MachineInterface for
`Advanced Teleoperation,” Science, vol. 208, No. 4450, pp.
`1327-1335, 1980.
`Bejezy et al., “Generalization of Bilateral Force-Reflecting
`Control Of Manipulators,” Proceedings of Fourth CISM-
`IFToMM, Sep. 8-12, 1981.
`Subsystem/Telerobot
`McAffee
`et
`al.,
`“Teleoperator
`Demonstrator: Force Reflecting Hand Controller Equipment
`Manual,” JPL 1988, JPL D-5172.
`Minsky, “Computational Haptics: The Sandpaper System
`for Synthesizing Texture for a Force-Feedback Display,”
`Ph.D. Dissertation, MIT, Jun. 1995, archived Jul. 6, 1995.
`Jacobsen et al., “High Performance, Dextrous Telerobotic
`Manipulator With I'orce Reflection,” /ntervention/ROV °91
`Conference & Exposition, Hollywood, Florida, May 21-23,
`1991.
`
`Shimoga, “Finger Force and Touch Feedback Issues in
`Dexterous Telemanipulation,” Proceedings of Fourth An-
`nual Conference on Intelligent Robotic Systems for Space
`Exploration, Rensselaer Polytechnic Institute, Sep. 30-Oct.
`1, 1992.
`IBM Technical Disclosure Bulletin, “Mouse Ball-Actuating
`Device With Force and Tactile Feedback,” vol.32, No. 9B,
`Feb. 1990.
`
`Terry et al., “Tactile Feedback In A Computer Mouse,”
`Proceedings of l’ourteenth Annual Northeast Bioengineer-
`ing Conference, University ofNew Hampshire, Mar. 10-11,
`1988.
`
`Howe, “A Force-Reflecting Teleoperated Hand System for
`the Study of Tactile Sensing in Precision Manipulation,
`Proceedings ofthe 1992 IEEE International Conference on
`Robotics and Automation, Nice, France, May 1992.
`Eberhardt et al., “OMAR—A Haptic display for speech
`perception by deaf and deaf-blind individuals,” JEFE Virtual
`Reality Annual International Symposium, Seattle, WA, Sep.
`18-22, 1993.
`tactile displays:
`“Multidimensional
`al.,
`Rabinowitz et
`Identification of vibratory intensity, frequency, and contrac-
`tor area,” Journal of The Acoustical Society ofAmerica, vol.
`82, No. 4, Oct. 1987.
`Bejezy et al., “Kinesthetic Coupling Between Operator and
`Remote Manipulator,” International Computer Technology
`Conference,
`The American
`Society of Mechanical
`Engineers, San Francisco, CA, Aug. 12-15, 1980.
`Bejezy et al., “A Laboratory Breadboard System For Dual-
`Arm Teleoperation,’” SOAR °89 Workshop, JSC, Houston,
`TX, Jul. 25-27, 1989.
`Ouhyounget al., “A Low-Cost Force Feedback Joystick and
`Its Use in PC Video Games,” IEEE Transactions on
`Consumer Electronics, vol. 41, No. 3, Aug. 1995.
`Marcus, “Touch Feedback in Surgery,” Proceedings of
`Virtual Reality and Medicine The Cutting Edge, Sep. 8-11,
`1994.
`
`Bejezy, et al., “Universal Computer Control System (UCCS)
`For
`Space
` Telerobots,” CH2413-3/87/0000/0318501.
`001987 IEEE, 1987.
`
`* cited by examiner
`
`6
`
`

`

`U.S. Patent
`
`Apr.11, 2006
`
`Sheet 1 of 25
`
`US 7,027,032 B2
`
`
`
`
`
`AUDIO OUT-
`
` SERIAL
`| GAME |
`
`PORT
`| PORT |
`
`
`
`
` SENSOR
`
`II
`
`
`
`1!| 1
`
`l
`
`L
`
`POWER
`
`|
`
`|!
`
`SUPPLY ~~ 40
`
`Gj
`
`J
`
`PUT DEVICE
`
`INTER-FACE
`
`
`
`FEEL
`—— |
`INPUT
`MANIP—|
`a eeee!
`SAFETY
`SWITCH
`ACTUATORS !
`30!
`41
`t—4----4-----—:saaehenietententententantententententmamiceseten 4
`
`1
`
`[
`
`7
`
`

`

`U.S. Patent
`
`Apr.11, 2006
`
`Sheet 2 of 25
`
`US 7,027,032 B2
`
`
`
`8
`
`

`

`U.S. Patent
`
`Apr.11, 2006
`
`Sheet 3 of 25
`
`US 7,027,032 B2
`
`8[iJShSnL
`WZenpS
`
`n
`
`174a
`
`9
`
`
`
`

`

`AAe907 doo
`
`
`
`JOUuodJSOL{
`
`ceAeyeq
`
`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 4 of 25
`
`US 7,027,032 B2
`
`
`
`
`
`O07JONUODje007
`
`
`
`sjeuBissosues
`
`¥YOSS300U"d
`
`sgoyenjoy\_r91
`
`syeubis
`
`9¢
`
`1 lo
`
`
`
`MOVEdass3OHOS
`
`bl
`
`“WdaAHdldad
`
`!11
`
`i
`
`es
`
`400dW907
`
`SPUBWLUOD89104
`
`11i111'I11
`
`ch
`
`LSOH
`
`YaLAdNOO
`
`10
`
`10
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 5 of 25
`
`US 7,027,032 B2
`
`vOEe
`
`ZuoyngguonngguoyWNg
`
`icyhe,
`[Al2]ELV)fs]eI=|]e](Ts)SI[ale(O
`
`
`
`
`djasmopui199/90upyaly
`
`Merameeis
`
`Meeeecsuoj|puoD@
`
`SOARM@Buds@eeu
`
`uo
`
`Ole
`
`yg18]u8D
`
`eI8U|
`
`Burls
`
`
`
`me][onUuDDOoj[pped
`
`
`
`,4ssaud‘djoy104
`
`olé
`
`OE
`
`OLE
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 6 of 25
`
`US 7,027,032 B2
`
`oa
`
`zuoyng|uoyng
`
`€uoyng
`
`Zuoyng9uoyngguoyngpuoyng
`
`QOQOOO000O0
`
`00°00!
`urer)
`
`
`
`djeHsmopuiAy99lqOupyaly
`
`ey
`le
`
`PE
`
`eeu©uolou
`
`Jedweq3SUONIPUCD@
`
`JUSIDIJE0D||sixyxX]sixy
`
`oce/Buds
`puegpeeq
`uoTeINeS
`
` 10}03/A,@dogWewsGdeemsoipoued&OIPOad®SOARM@GE
`someukq@9010-4
`
`-Jajdayo>
`
`BPE
`
`SHO
`
`yuq4aque5
`
`a|Ppped
`
`Bus
`
`
`
`MB"T[OUOD
`
`1}000u
`
`yoreduy}
`
`pinbry
`
`BIUSU
`
`
`
`L4ssoud‘djay304
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Apr.11, 2006
`
`Sheet 7 of 25
`
`US 7,027,032 B2
`
`344
`
`340
`
`338
`
`+> 342
`
`236
`350
`
`Gig, 7@
`
`338
`
`350
`
`344
`
`354
`
`(342 fo
`>
`350
`
`336
`
`340 a 7}
`
`344
`
`340
`
`338
`
`350
`
`342
`
`hig, 7C
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Apr.11, 2006
`
`Sheet 8 of 25
`
`US 7,027,032 B2
`
`360
`
`378 3
`
`374
`
`372
`
`14
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 9 of 25
`
`US 7,027,032 B2
`
`0ge
`
`Pee
`
`907
`
`66€OL
`
`000000'1|poued
`
`
`
`14sseid‘djepy105
`
`
`
`
`SZS°S16|feAlequlseBbuy|00001:98yeedoy
`umetaoOO'OOr]YEOO'OOT]19407
`
`
`
`UoHNdIaJIGedojeauyq—epnyubeyy
`
`88EvorcorOIPOLeed
`
`
`
`
`[Ale]SLB1E}fsleiepole][ETS]fois][(ofeiO)
`[007001ures|0000"1)uoneung[]00001
`apesERYsoumeuka©90044
`
`Cooorsy/[o00ro]oun
`
`
`ro}010}P=aanSL)PICO@ah
`
`diesmopui,wolqdupyZey
`
`Merane)oyne
`
`
`
`miot=)lelaMeese)
`
`
`
`deemsolpoued&
`
`J0JO8/A,0dogyewsGo
`
`cO0e
`
`15
`
`15
`
`
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 10 of 25
`
`US 7,027,032 B2
`
`ey)saneo>me)
`(Alals]SS(Ble) djeHsmopulm
`
`phe|(eoroe](LETC
`|LaWANTE
`
`
`000000'f]poueg|yensojuygop_2288uLyeedoy
`wes||SrZzjules|0000}
`[Ale]aleledfs]eilefale]
`[cocoon|[aT=StenS|
`
`
`LOPGEr'O}/[OL1760'0}ouny
`
`apesyOENY
`Ze6S6056]1027
`
`re]ste)¢1-YaPelaX=)TeoD)
`
`dgaMSIPOLUed
`Jedweq6onessuonipucd@
`Solpoued@oIpoLedi)
`yeladupyely
`
`
`
`uondaq
`
`16
`
`sojweuig@80104J0}E/,o
`
`16
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 11 of 25
`
`US 7,027,032 B2
`
` L4ssoid
`
`
` so1weukg@yneyequojIpuoDsixypuosesUOHIPUCDSPXISUswoalqodio4ee9SPLOPseduegq|vce\,senem@PEToNU-Tapererecad)dSUuOCHIPuOy@yooued°punodwoo@))mae
`
`
`
`BANISOgSAeboN2S¢7Oamisod SAVEDONQ/p
`
`
`JesyOpuegpeegsexyyogl]||/1esHOpuegpeeqsexyyog[4]_0
`
`0000Oooo|uojyeinjesgogool|uoReinyes
`
`
`
`
`ee0001}weroyeco/Y4689]cool]we!oyeod
`0O|oujewuwAs[]oUJEWLAS5[sez]
`
`
`
`
`
`
`
`90f°DpeouenpylejduispeoueapyledwisI~OSPSOWOACL
`
`Ale]GLEfaleiseio[e]Ss)ESle!
`
`208
`
`deysmopuiy,welqGupdeit
`
`LS?
`
`17
`
`8S7
`
`09P
`
`‘djo}y404
`
`
`
`
`
`LpennUN-OIPNISFOWOsICD
`
`17
`
`
`

`

`MCESSEToleOemLOoy
`
`yneeqdodweq
`
`
`
`
`JSHOpuegpeeqsexyuylog0uoweines}}|__0]omauiwikg9OjoujeuwAg[0002|JESYOpuegpeeqsexyyOog
`
`
`UORIPUODpsauenpylajdwispeoueapy[ajdwis|UOIIPUodSIXPuoveSSIXYISIjaSIXXX}1%001
`
`
`
`
`
`qua|oyjecgderesOPGS| jualayje0D
`
`00001]00001] uoHeimes
`
`
`
`[Ale]alefete]ei=fale](Bel)Sls][ole(O
`
`Jadweg::|payun>
`
`
`dieaH™smopuinyelggupyony
`someukg@VEsenem@SUOHIPUDD@sedureq3punodwion@)
`
`
`
`
`
`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 12 of 25
`
`US 7,027,032 B2
`
`
`
`SAIISOgeAneben
`
`vor
`
`SOxyured
`
`srrelacary
`
`SoWOAel@spalgodio&®
`
`18
`
`Ldsseud‘djayyyoy
`
`18
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 13 of 25
`
`US 7,027,032 B2
`
`
`
`SOUPRADYja;dw
`
`il
`
`Sexyules)
`
`Jadweq
`
`ICD
`
`0OjoujowuwAsTl]v6P8Lb|oujewwAsC][000s]
`
`
` egmee) Ale]Le)feletseole](Blais)Sseo
`
`eanebaneANSOgsAVeEbeN
`
`
` 00001|QOOl| ywaloyjeogLOGILOL}|jualoiyeod
`
`
`
`
`
`
`UOIHPUucydSIXyPuodVEeS
`
`pooueapy|eldwisg
`uoHeunyes0929Z61L9UOHeINeS
`
`
`
`}SYOpuegpeegsexyyog[_]}_0
`
`
`sop”peouenpy[erduas
`elb[aSY]|2001x
`
`
`UOHIPUODSIXyISJ14
`ptclellSicTqbeam=)1bgear)
`
`dey”smopulntyelqGpyoy|pannun-olpms
`
`
`SOHOS
`
`Ljsseud‘diay404
`
`sauoAej@spalqodin@soiueukg@serem@SUOMPUOD@
`
`19
`
`19
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 14 of 25
`
`US 7,027,032 B2
`
`elamecaty
`1SYOpuegpeeqsexywogl]
`
`
`0OjoujeuwAS[J
`
`UOHIPUOTSIXPUOSES
`
`
`Ale]GE)Blotseiole]Fa]SSBem
`
`sexyyiog[]909¢26721|omewwAs[}[00°0s|JesyOpuegpeaq
`
`soAeM@Treaeean)SUOIPUOD@Jedweq3.
`spalqodio®urepsoueukg@Jedweqg
`
`
`die”smopuiy™yosiqdupzelt
`
`
`iak=)PUPSMeme)|°LA}eenFOtas@on
`
`waroyeoy
`
`uoneinyes
`
`jUa!D1Ye0D
`
`uorenes
`
`
`
`Lasseid‘djayu0g
`
`
`
`peourapy|ejdwis
`
`sejoae4@)
`
`ay
`
`20
`
`punodwog©)
`
`20
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 15 of 25
`
`US 7,027,032 B2
`
`[ce]7|
`
`AON
`
`Buds SUIFToISHMnelaRcoe)
`yesHOpuegpeegsexyYog(]
`oO]oJomoauuAS[]
`OCOCcOL|uolzeinjzes
`(29
`
`
`SIXYISUfaSR]b6rUORIPUOD|se001|—957,
`
`19Spuegpeeg
`
`
`a,peoueapylejduns|rt«~~~26h
`00001]00001|860E
`
`1996}
`
`
`oujeuuAs[7][se'96]
`sexyyjog0udlEanyes
`
`io[TO]ILI]e|
`cadesedureq
`
`qugOyjaod
`Buuds
`
`diaysmopuiyyHalaupoy
`menamec)
`
`MUlaeelsee):OnmLee
`IS]lole!O
`queun9yeBuds@Budsrs]juaingyeadojs@adols@eno
`
`ejbuyeseuegesuy Buudsajbuy©edojs
`
`
`LJsseud‘djey}]40-4
`
`remsxv@s19IUegSy©enejouy
`spelqodid®soweuig@
`JadweqSad
`SUOHIPUCD@punodwoo@)
`
`00€
`
`867
`
`EE)
`
`SOxurer)
`
`
`
` BAIISOgeAlebeN
`q96r_967
`
`OS
`
`Zerg]
`
`uoHoUs
`
`21
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 11, 2006
`
`Sheet 16 of 25
`
`US 7,027,032 B2
`
`
`
`BANISOdSAleben
`
`00$
`
`|
`
`Ok]11)00)meme2/oslotO)oOmmLGD)
`
`JesyOpuegpeegsexyyog[_]
`000000001] uoHeinjes
`
`
`00oLQ00L|ualaye0D
`
`0OjoujowwAs(]
`UONIPUODSIxyPUdDES
`
`
`
`puegpeeqsexyyjog[]}_0€SZ