(19) United States
`(2) Patent Application Publication (10) Pub. No.: US 2007/0279392 A1
`Rosenberg et al.
`(43) Pub. Date:
`Dec. 6, 2007
`
`US 20070279392A1
`
`(54) NETWORKED APPLICATIONS INCLUDING
`HAPTIC FEEDBACK
`
`(76) Inventors: Louis B. Rosenberg, San Jose, CA
`(US); Scott B. Brave, Edwards, CO
`(US); Sian W. Tan, Mountain View, CA
`(US)
`Correspondence Address:
`PATENT DEPARTMENT (51851)
`KILPATRICK STOCKTON LLP
`1001 WEST FOURTH STREET
`WINSTON-SALEM, NC 27101 (US)
`(21) Appl. No.:
`10/615,927
`
`(22) Filed:
`
`Jul. 10, 2003
`Related U.S. Application Data
`(63) Continuation of application No. 09/153,781, filed on
`Sep. 16, 1998, which is a continuation-in-part of
`application No. 09/050,665, filed on Mar. 30, 1998,
`now Pat. No. 6,219,033, which is a continuation of
`application No. 08/566,282, filed on Dec. 1, 1995,
`now Pat. No. 5,734,373, and which is a continuation
`of application No. 08/571,606, filed on Dec. 13, 1995,
`now Pat. No. 6,219,032, and which is a continuation
`of application No. 08/691,852, filed on Aug. 1, 1996,
`now Pat. No. 5,956,484, and which is a continuation
`
`of application No. 08/664,086, filed on Jun. 14, 1996,
`now Pat. No. 6,028,593.
`(60) Provisional application No. 60/017,803, filed on May
`17, 1996.
`
`Publication Classification
`
`(51) Int. Cl.
`(2006.01)
`G{}6F 3/04 I
`(52) U.S. Cl. ................................................. 345/173; 463/1
`
`ABSTRACT
`(57)
`Method and apparatus for providing peer-to-peer force feed
`back over a computer network. A network force feedback
`system includes a network, a first computer coupled to the
`network, and a second computer coupled to the network. The
`first and second computers each include a visual display and
`a force feedback interface device. Each computer provides
`a force feedback signal to its force feedback device based on
`information received from the other, remote computer and in
`some cases also based on input from the local force feedback
`device. Positional information of each force feedback device
`and/or feel sensation information can be transmitted
`between the computers over the network. A graphical envi
`ronment can be displayed to enhance the interaction between
`users. The present invention therefore permits two computer
`users to interact using force feedback provided over a
`network on a peer-to-peer basis.
`
`Location |http://www.immerse.com/demo
`L-1 C-J Y238
`[T]
`
`244
`
`
`
`246
`
`228
`
`D.
`240
`
`J. ==
`
`236 4-) 242
`
`2-250
`
`254
`
`w
`
`:
`
`SPRING
`
`248
`
`KEEP
`OUT
`
`FORCE TO LEFT
`
`
`
`
`
`264
`
`266
`
`268
`
`
`
`232
`Tº
`
`.
`
`-
`
`Immersion Ex 2004-1
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 1 of 13
`
`US 2007/0279392 A1
`
`10~
`
`
`
`WEB
`SERVER
`MACHINE
`
`--~18
`
`28
`
`11 || /
`?cLENTI --14
`*]
`
`30
`
`12
`| . CLIENT
`e.
`
`24
`
`-
`
`.
`
`26
`
`FFD
`
`FFD
`
`Immersion Ex 2004-2
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 2 of 13
`
`US 2007/0279392 A1
`
`GO
`CY)
`
`go É
`Co
`is
`Lil
`>
`
`§
`S.
`
`Immersion Ex 2004-3
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 3 of 13
`
`US 2007/0279392 A1
`
`PERSONAL compUTER systEM 2-48
`
`| as smº, sm sº sº - me = * = ** * * * * * * * * * * = * * * * * * * * * * * * * * * *
`|
`!
`. .
`
`62
`
`-
`SYSTEM
`CLOCK
`
`AUDIO OUT-
`PUT DEVICE
`
`,-46
`-
`
`HEAR
`56
`
`—l
`
`-
`
`-
`
`T
`
`CLK
`
`66
`
`36
`
`64
`
`PROCESSOR —- DEVICE ||
`
`|
`
`MICRO- ºf ?ºft-i-ºw
`
`-
`
`t
`
`-
`
`*
`
`|
`
`
`
`
`
`
`
`
`
`
`
`FORCE FEEDBACK
`DEVICE
`
`
`
`?sensor
`INTER
`FACE
`
`80
`ACTUATOR I->
`INTERFACE |
`
`* = ** = = = = * ~ * *-* = = * * =
`
`A C T U A T O R S
`
`Immersion Ex 2004-4
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 4 of 13
`
`US 2007/0279392 A1
`
`
`
`figure 4a
`
`172
`122
`128
`%3&
`SSSSSSSSSSSSSSSSSSSSSSSSSSSS
`
`
`
`120
`
`112
`
`122
`
`126
`
`Immersion Ex 2004-5
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 5 of 13
`
`US 2007/0279392 A1
`
`
`
`Immersion Ex 2004-6
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 6 of 13
`
`US 2007/0279392 A1
`
`SERVER MACHINE
`PROCESSORRUNNING THE |…....................................
`HTTP SERVER SOFTWARE,
`CONNECTED TO INTERNET
`
`18
`
`14
`
`
`
`
`
`10
`
`-
`:
`DATA
`HTML FILE .
`:
`RECUEST|
`IFF FILE
`:
`CLIENT MACHINE
`tº
`:
`TCP/IP
`: "...I. ~ (..…”.”
`PROCESSOR RUNNING
`CoNNEC. ...ADDITIONAL CLIENTS.;
`INTERNET BROWSER SOFT. I for :* :
`WARE AND FORCE FEEDBACK I-............x-
`:
`| DRIVER SOFTWARE, THIS
`:
`:
`PROCESSOR is connecTED
`:
`:
`TO INTERNET AND CONNEC-
`:
`:
`TED TO FORCE FEEDBACK I
`------------------1- …
`DEVICE
`:
`
`
`
`24
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`:
`TRACKING AND BUTTON DATA
`:
`FORCE FEEDBACK COMMANDS
`K DEV
`26 : "...[. ~f~,
`-
`FORCE FEEDBAC
`ICE
`*
`: ADDITIONAL FORCE
`i. FEEDBACKDEVICES...;
`THIS DEVICE HAS SENSORS
`AND ACTUATORS SO IT CAN
`TRACKAUSER's MOTION,
`MONITOR BUTTON PRESSES,
`AND PROVIDE OUTPUT FORCE
`FEEDBACK SENSATIONS, IT
`WILL SEND TRACKING AND
`SENSOR INFORMATION TO :
`CLIENT. IT WILL RECEIVE
`FORCE FEEDBACK COMMANDS
`FROM CLIENT.
`
`Immersion Ex 2004-7
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 7 of 13
`
`US 2007/0279392 A1
`
`148
`
`146
`/T
`
`
`
`
`
`SEND COMMAND
`REQUEST TO HOST
`OF URL
`
`RECEIVE
`HTML FILE
`
`150
`-
`
`-
`
`152
`
`
`
`
`
`OBTAIN HTML
`COMPONENT .
`
`154
`
`156
`YES
`
`
`
`NO
`
`@Dº"
`
`PARSE AND INTERPRET
`HTML COMPONENT
`
`- -160
`
`
`
`
`
`
`
`
`
`
`
`
`
`- Jigure 7a
`
`Immersion Ex 2004-8
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 8 of 13
`
`US 2007/0279392 A1
`
`-
`170–K
`
`>
`
`-
`
`-CENTER-168
`<EMBED "FORCEBUTTON.IFF; ~172
`SIZE = 100,100-174
`INITSTATE = UP-176
`F160
`FORCEEFFECT = VIBRATION(PARAMS)-173
`TRIGGER = MOUSEWITHIN(PARAMS) +
`BUTTONSTATE-Sliga
`|CON = LOUIS.G|F-184
`TEXT = "HI, I'M LOUIS"—186
`FONT = HELVETICA-139
`</CENTERS-190
`: }~le
`
`</BODYS-194
`</HTMLST-196
`
`
`
`
`
`
`
`FOR (INT is 0; i < 5; ii.1)
`VIBRATE(PARAMS);
`WAIT 5;
`-
`
`200
`_-198
`
`202
`
`204
`
`- Jigure 76
`
`Immersion Ex 2004-9
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 9 of 13
`
`US 2007/0279392 A1
`
`Gº
`
`2O6
`
`160
`
`/T
`
`
`
`
`
`
`
`
`
`EMBED
`TAG TO FORCE
`OBJECT
`
`PLUG-IN
`INTERPRETS
`..IFF FILE
`
`210
`.
`
`
`
`| No
`
`ºlº ?igures
`
`212
`
`218
`
`-220
`
`
`
`
`
`BUILD FORCE OBJECT
`BASED ON NAME/VALUE
`PAIRS
`
`222
`
`224
`
`Immersion Ex 2004-10
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 10 of 13
`
`US 2007/0279392 A1
`
`Location http://www.immerse.com/demo
`CTI ETT, CT, LTJ CT, CT, 338
`
`TEXTURE
`
`VISCOSITY
`-
`
`242
`
`228
`
`
`
`FORCE TO LEFT
`-
`
`.
`
`(TEXTURE;
`
`.
`W
`2'
`"s
`%fixto:
`262
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`-
`
`/-264
`
`266
`
`2268
`
`232
`ºf
`
`figure 9a
`
`Immersion Ex 2004-11
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 11 of 13
`
`US 2007/0279392 A1
`
`270s,
`
`MONITOR
`POSITION AND
`BUTTON STATE
`
`~274
`
`CREATE FORCE
`FEEDBACK COMMAND
`IN RESPONSE TO
`POSITION AND STATE
`
`
`
`276
`
`SEND COMMAND
`TO DLL TO PUT
`MEDIUM LEVEL
`COMMAND ON
`INTERFACE
`
`
`
`278
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`figure 10
`
`Immersion Ex 2004-12
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 12 of 13
`
`US 2007/0279392 A1
`
`
`
`
`
`
`
`-T
`‘Y
`
`Ll
`
`Immersion Ex 2004-13
`Apple v Immersion
`IPR2016-01372
`
`

`
`Patent Application Publication Dec. 6, 2007 Sheet 13 of 13
`
`US 2007/0279392 A1
`
`
`
`FIG, 12B
`
`Immersion Ex 2004-14
`Apple v Immersion
`IPR2016-01372
`
`

`
`US 2007/0279392 A1
`
`Dec. 6, 2007
`
`NETWORKED APPLICATIONS INCLUDING
`HAPTIC FEEDBACK
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`[0001] The present application is a continuation-in-part of
`U.S. patent applications:
`[0002] Ser. No. 09/050,665, entitled “Method and Appa
`ratus for Controlling Force Feedback Interface Systems
`Using a Host Computer”, filed Mar. 30, 1998, which is a
`continuation of U.S. Pat. No. 5,734,373, filed Dec. 1, 1995;
`[0003] Ser. No. 08/571,606, entitled “Method and Appa
`ratus for Providing Force Feedback for a Graphical User
`Interface,” filed Dec. 13, 1995;
`[0004] Ser. No. 08/691,852, entitled “Method and Appa
`ratus for Providing Force Feedback over a Computer Net
`work,” filed Aug. 1, 1996; and
`[0005] Ser. No. 08/664,086, entitled “Method and Appa
`ratus for Providing Simulated Physical Interactions within
`Computer Generated Environments,” filed Jun. 14, 1996,
`which claims priority of provisional application Ser. No.
`60/017,803, filed May 17, 1996,
`[0006] all of which are incorporated herein by reference
`for all purposes.
`
`BACKGROUND OF THE INVENTION
`[0007] This invention relates generally to human/com
`puter interfaces, and more particularly to human/computer
`interfaces with force feedback that can operate over a
`network.
`[0008] Computer networks have become essential to allow
`users and computers to communicate with each other. Users
`transmit and receive data over networks in offices, at home,
`or in portable devices, and do so for a variety of tasks and
`applications, including communication, distribution, and
`entertainment. Many different types of networks are used.
`Local Area Networks (LANs) are typically provided in a
`limited area and include a relatively small number of com
`puter nodes. The most large scale example of a network is
`the Internet, which has become extremely popular. The
`Internet is a Wide Area Network (WAN) that is structured
`and distributed such that no one authority or entity manages
`the network. Different communication protocols can be used
`over networks to allow computers to communicate with each
`other; for example, protocols such as “Transmission Control
`Protocol/Internet Protocol” (TCP/IP) and the “World Wide
`Web” (WWW) are used over the Internet. TCP/IP sends
`“packets” of data between a host machine, e.g. a server
`computer on the Internet, and a client machine, e.g. a user’s
`personal computer connected to the Internet, or between two
`client machines. The WWW is an Internet interface protocol
`which is supported by the same TCP/IP transmission pro
`tocol. Intranets are private networks based upon Internet
`standards, and have become quite common for managing
`information and communications within an organization.
`Intranets, since they adhere to Internet standards, can often
`use the same interface software as are used on the Internet,
`such as a web browser.
`[0009] A variety of information is currently transferred
`over computer networks. For example, visual data, text data,
`
`and sound data can be transmitted over the Internet and the
`WWW. Image data can be sent from one client machine to
`another (or from a server machine to a client machine) in a
`variety of formats. Or, for example, data packets coded in
`TCP/IP format can be sent from one client machine to
`another over the Internet to transmit sound data. This
`last-mentioned technique forms the basis for Internet tele
`phony.
`[0010] While the transmission of visual images (both
`static and dynamic, i.e. video), text, and sound over net
`works, such as the Internet, is well-known, the transmission
`of other types of sensory data has not been well explored. In
`particular, the transmission of data over networks pertaining
`to the sense of touch and/or force has not been established.
`“Force feedback” allows a user to experience or “feel”
`tactile sensations as provided through computational infor
`mation. Using computer-controlled actuators and sensors on
`a force feedback device, a variety of realistic sensations can
`be modeled and experienced by the user. This useful and
`highly immersive sensory modality for interacting with the
`Internet and other users over the Internet has hereto been
`unavailable.
`
`SUMMARY OF THE INVENTION
`[0011] The present invention is related to the transmission
`of information pertaining to a subset of the sense of touch,
`i.e. the transmission of forces to a user over a computer
`network system. The “force feedback” provided by the
`methods and apparatus of the present invention enhance the
`sensory experience of user-to-user interactions to provide a
`richer, more interesting, and more enjoyable experience.
`[0012] A network force feedback system in accordance
`with the present invention includes a network, a first com
`puter coupled to the network, and a second computer
`coupled to the network. The first and second computers each
`include a visual display and a force feedback interface
`device. The interface device is capable of providing a
`computer input to the computer and also includes an actuator
`to output force feedback to a user in response to a force
`feedback signal provided by the computer. At least one of
`the computers develops an image on the visual display that
`is associated with stored force feedback information, and
`produces the image and the force feedback signal based on
`information received from the other, remote computer. Pref
`erably, the computers produce the images and the force
`feedback signals based on information received from the
`remote computer and based on the computer input from the
`local force feedback device. The force feedback device can
`include a local microprocessor that communicates with the
`computer such that the force feedback signal can take the
`form of a relatively high-level force command. The present
`invention therefore permits two computer users to interact
`using force feedback provided over a network on a client
`to-client (peer-to-peer) basis.
`[0013] A method for providing force feedback between
`two computers over a network includes establishing a con
`nection between a first computer and a second computer
`over a network, sending first computer information to a
`second computer from the first computer over the network,
`and providing a force feedback signal to the second force
`feedback device from the second computer, where the force
`feedback signal is based on the first computer information.
`
`Immersion Ex 2004-15
`Apple v Immersion
`IPR2016-01372
`
`

`
`US 2007/0279392 A1
`
`Dec. 6, 2007
`
`The force feedback signal causes the second force feedback
`device to output forces to the second user using an actuator
`of the force feedback device. Similarly, second computer
`information is sent to the first computer from the second
`computer over the network, and a force feedback signal is
`provided to the first force feedback device from the first
`computer. The force feedback signal is based on the second
`computer information, where the force feedback signal
`causes the first force feedback device to output forces to the
`first user using an actuator of the first force feedback device.
`The force feedback signal can also be based on the input
`information from the force feedback devices. The informa
`tion sent over the network can include position information
`describing a position of a manipulandum of the force
`feedback devices, and/or can include force feedback infor
`mation indicating a force sensation to be output by the
`remote force feedback device. The computers can each
`display a graphical environment having a first graphical
`object controlled by the first user and a second graphical
`object controlled by the second user.
`[0014] In a different aspect, a method is disclosed of
`allowing two users to interact physically over a computer
`network, wherein a first manipulandum is physically con
`tacted and moved by a first user in at least one degree of
`freedom and a second manipulandum is physically contacted
`and moved by a second user in at least one degree of
`freedom. First information is transmitted over the computer
`network, including an indication of movement of the first
`manipulandum, to a second manipulandum physically con
`tacted by a second user. A force is applied to the second
`manipulandum based on the indication of movement of the
`first manipulandum such that the second user feels an
`interaction based on movement of the first manipulandum.
`Second information is similarly transmitted to the first
`manipulandum such that the first user feels an interaction
`based on movement of the second manipulandum. Two users
`can thus physically exchange information and interact over
`a computer network.
`[0015] The present invention adds a new sensory modality
`when interacting with a networked computer system. More
`particularly, force information can be either downloaded to
`a client machine from a server machine connected to the
`network, or force information can be passed between two or
`more client machines on the network. Peer-to-peer or server
`to-peer direct interaction allows two or more users to
`interact using a different sensory modality, the sense of
`touch. The interaction may be subject to some transmission
`(“latency”) delays on networks such as the Internet, but
`permits remote interactivity with a client’s force feedback
`device in new ways to enhance communication and inter
`action between users.
`[0016] These and other advantages of the present inven
`tion will become apparent upon reading the following
`detailed descriptions and studying the various figures of the
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`[0017] FIG. 1 is a pictorial representation of the Internet,
`a web server machine, and two client machines;
`[0018] FIG. 2 is a block-diagram of a client machine used
`in the present invention;
`
`[0019] FIG. 3 is a block-diagram of a force feedback
`system in accordance with the present invention;
`[0020] FIG. 4a is a perspective view of a preferred human/
`computer interface (“force feedback device”) of the present
`invention;
`[0021] FIG. 4b is a cross-sectional view taken along line
`4b-4b of FIG. 4a:
`[0022] FIG. 5a is a perspective view of another preferred
`embodiment for a force feedback device in accordance with
`the present invention;
`[0023] FIG. 5b is a perspective view of a first alternate
`embodiment for the force feedback device of FIG. 5a;
`[0024] FIG. 5c is a perspective view of a second alternate
`embodiment of the force feedback device of FIG. 5a;
`[0025] FIG. 6 is a block diagram of a wide area network
`(WAN) based upon Internet TCP/IP protocol and supporting
`World Wide Web (WWW) HTML protocols in accordance
`with the present invention;
`[0026] FIG. 7a is flow-diagram of a “Acquire URL"
`process in accordance with the present invention;
`[0027] FIG. 7b is an example of an HTML file of the
`present invention sent from a web server machine to a client
`machine;
`[0028] FIG. 8 is a flow-diagram of the “Parse and Interpret
`HTML Component” step of FIG. 7a;
`[0029] FIG. 9 is a flow-diagram of the “Plug-in Interpret
`IFF File” step of FIG. 8:
`[0030] FIG. 9a is an illustration of an image displayed on
`a visual display of a client computer as generated from a
`downloaded HTML web page file;
`[0031] FIG. 10 is a flow-diagram of a process for moni
`toring the “pointer state” of the force feedback device of the
`present invention;
`[0032] FIG. 11 is a schematic diagram of a multi-computer
`network system used in accordance with client-to-client
`embodiments; and
`[0033] FIGS. 12a and 12b are diagrammatic illustrations
`of a client-to-client game embodiment displayed on a dis
`play screen of a client computer.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`[0034] In FIG. 1, a network system 10 includes a network
`and a number of computers or “machines” coupled to the
`network. In the described example of FIG. 1, the network is
`the Internet 12. For example, a first client machine 14, a
`second client machine 16, and a web server machine 18, are
`coupled to the Internet 12. Although embodiments specific
`to the current form of the Internet are described, it should be
`appreciated that the present invention can also be used in
`conjunction with many different types of networks (any
`LAN or WAN) using appropriate communication protocols.
`Such types of networks and the communications used over
`such networks are well known to those skilled in the art.
`[0035] As noted previously, both the Internet 12 and
`Intranets operate using the same TCP/IP protocols. This
`
`Immersion Ex 2004-16
`Apple v Immersion
`IPR2016-01372
`
`

`
`US 2007/0279392 A1
`
`Dec. 6, 2007
`
`allows Intranets to use similar or the same server machine
`software and client machine software as are used in Internet
`12 applications. Therefore, it will be apparent to those
`skilled in the art that the following descriptions apply
`equally well to Internet, Intranet, and other forms of network
`systems that are compatible with the processes and appara
`tus disclosed herein.
`[0036] The Internet 12 includes a number of nodes 20 that
`are interconnected by data transmission media 22. These
`nodes are typically routers, switches, and other intelligent
`data transmission apparatus which route “packets” of TCP/
`IP information to the desired destination. In some instances,
`the nodes 20 comprise an Internet service provider (ISP) 20a
`which allows a client machine to access the “backbone” of
`the Internet. Alternatively, client machines and web servers
`can be coupled directly into the backbone of the Internet.
`[0037] As noted previously, the present invention is
`directed to the implementation of force feedback over a
`network, such as the Internet 12. To provide a user of a client
`machine with the experience of force feedback, force feed
`back human/computer interfaces (hereafter “force feedback
`devices”) 24 and 26 can be provided as part of the client
`machines 14 and 16, respectively. The client machines 14
`and 16 are typically provided with computer video monitors
`28 and 30 (which is one example of a “visual display”),
`respectively, which can display images I1 and I2, respec
`tively. Preferably, forces developed by force feedback
`devices 24 and 26 are correlated with the images I1 and I2
`of the client machines 14 and 16, respectively.
`[0038] The machines 14-18 are considered, in the lan
`guage of the Internet, to be “resources,” and each has its own
`unique Uniform Resource Locator or “URL.” In one
`embodiment of the present invention, a client machine, such
`as client machine 14 or 16, sends a request for a “web page”
`residing on, for example, web server machine 18. This is
`accomplished by the client machine sending a connection
`request and a URL which specifies the address of the web
`page to the web server machine 18. The web server machine
`18 then sends a web page 32 written in HTML format back
`to the requesting client machine where it is “cached” in the
`memory (typically the RAM, hard disk, or a combination of
`the two) of the client machine. In this embodiment of the
`invention, the image on the video display of the client
`machine is generated from the HTML web page file cached
`on the client machine, and force feedback is provided to a
`user through the force feedback device as he manipulates a
`user manipulable object of the force feedback device.
`[0039] In a peer-to-peer aspect of the present invention, a
`first client machine, such as client machine 14, and a second
`client machine, such as client machine 16, directly commu
`nicate force feedback commands to each other in standard
`TCP/IP protocol over the Internet 12. More particularly,
`client machine 14 can send force feedback and other infor
`mation to the URL of the client machine 16, and the client
`machine 16 can send force feedback and other information
`in standard TCP/IP packets to the URL of the client machine
`14. In this way, users of client machine 14 and client
`machine 16 can interact physically over the Internet 12. Of
`course, a server machine 18 can likewise directly commu
`nicate force feedback commands to a client machine 12 or
`14, or all three machines can interact. The client machines
`can also communicate directly over other types of networks
`
`and/or using other communication protocols. Peer-to-peer
`(i.e. client-to-client) communication is described below with
`reference to FIG. 11.
`[0040] In FIG. 2, a “personal” computer 34 architecture
`that can be used for client machine 14 or client machine 16
`is shown in block diagram form. It should be noted that a
`variety of machine architectures can be used to access the
`Internet 12, i.e. can be used as “network access computers.”
`The particular architecture shown for the computer 34 is a
`typical personal or “PC” computer architecture, such as that
`used with IBM compatible personal computers. Web server
`machines can also have similar architectures, but are often
`more powerful computers known as “workstations” that
`operate under some variant of the UNIX® operating system.
`The Internet service providers 20a are likewise often UNIX
`based computers or powerful personal computers running
`Windows NTR). The nodes 20 are most commonly routers
`built by Cisco Systems of San Jose, Calif. Client machine 14
`or 16 can also take other forms, such as a television
`including or connected to a microprocessor for Internet
`access, or a video game console system such as those
`available from Nintendo, Sega, or Sony. Force feedback
`devices used with such client machines can be appropriate
`for the particular embodiment, e.g., a TV remote control
`used for internet browsing on the abovementioned television
`can include force feedback functionality.
`[0041] The personal computer system 34 includes a micro
`processor 36 clocked by a system clock CLK and which is
`coupled to a high speed or memory bus 38 and to a lower
`speed or I/O bus 40. The system RAM 42 and ROM 44 are
`typically coupled to the high speed memory bus, while
`various peripherals, such as the video display, hard disk
`drive, Internet interface (often either a modem or an Ethernet
`connection), and force feedback device, are typically
`coupled to the slower I/O bus. The microprocessor executes
`programs stored in the various memories (RAM, ROM, hard
`disk, etc.) of the personal computer 34 to control, for
`example, the image display on the video display and the
`forces provided by the force feedback device. The manu
`facture and use of personal computers, such as personal
`computer 34, are well-known to those skilled in the art.
`[0042] In FIG. 3, a client machine 46 in accordance with
`the present invention includes a personal computer system
`48 and a force feedback human/computer interface or “force
`feedback device”50. A user 52 can receive visual informa
`tion 54 and auditory information 56 from the personal
`computer 48 and can manipulate the force feedback device
`50 as indicated at 58a and 58b to provide input, e.g., to
`command a cursor location on a visual display or other
`provide other control information. In addition, the user 52
`can receive force feedback 60 from the force feedback
`device 50 to represent physical “feel” sensations.
`[0043] The personal computer system 48 includes the
`microprocessor 36, the system clock 62, a video monitor 64
`(which is one type of “visual display”), and an audio device
`66. The system clock 62, as explained previously, provides
`a system clock signal CLK to the microprocessor 36 and to
`other components of the personal computer system 48. The
`display device 64 and the audio output device 66 are
`typically coupled to the I/O bus 40 (not shown in this figure).
`[0044] In this preferred embodiment, the force feedback
`device 50 preferably includes a local microprocessor 68, a
`
`Immersion Ex 2004-17
`Apple v Immersion
`IPR2016-01372
`
`

`
`US 2007/0279392 A1
`
`Dec. 6, 2007
`
`local clock 70, optional local memory 71 for the local
`microprocessor 68, a sensor interface 72, sensors 74, a user
`manipulatable object 76, “other” input interface 78, an
`actuator interface 80, a safety switch 82, and actuators 84
`which provide a force F to the object 76, and an optional
`power supply 86 to provide power for the actuator interface
`80 and actuator 84.
`[0045] The microprocessor 36 of the personal computer
`system 48 is coupled for communication with the local
`microprocessor 68 of the force feedback device 50. This
`communication coupling can be through a serial port cou
`pling 88 to the personal computer system, or through a game
`port coupling 90 to the personal computer system. Virtually
`all personal computer systems built to the IBM PC/AT
`standards will include a serial port and a game port. As
`noted, the serial port will permit two-way communication
`between microprocessor 36 and microprocessor 38, and thus
`is preferable over the game port coupling which only
`permits one-way communication from the local processor 68
`to the microprocessor 36. In consequence, a serial port
`connection between the personal computer system 48 and
`the force feedback device 50 will permit force feedback
`commands to be sent from the microprocessor 36 to the local
`microprocessor 68, while a game port connection alone will
`not be able to provide this function. However, some simpler
`forms of “reflex” type force feedback can still be provided
`by the force feedback device 50 under the control of the
`local microprocessor 68 even if only a slower interface is
`used. It should also be noted that the microprocessor 36 and
`a local microprocessor 68 may communicate over both the
`serial port and game port connection to provide a greater
`communication bandwidth. A preferred serial port is the
`Universal Serial Bus (USB) of a personal computer,
`although an RS-232 serial bus, or other serial busses, a
`parallel bus, an Ethernet bus, or other types of communica
`tion links can also be used.
`[0046] In use, the user 52 of the client machine 46 grasps
`the user object 76 (or “manipulandum”) of the force feed
`back device 50 and manipulates (i.e. exerts a force to move
`or attempt to move) the user object to cause a “pointer” or
`other graphical object to move in the image displayed by the
`display device 64. For example, a pointer typically takes the
`form of a small arrow, a pointing hand, or the like. The
`sensor 75 senses the movement of the user object 76 and
`communicates the movement to the local microprocessor 68
`through the sensor interface 72. The local microprocessor 68
`then communicates through serial port 88, game port 90, or
`both to the microprocessor 36 to cause the microprocessor
`36 to create a corresponding movement of the pointer on the
`image displayed upon the visual display 64. In some
`embodiments, the sensors 74 can communicate directly to
`microprocessor 36 without the use of local microprocessor
`68. The user can also create other input, such as a “button
`click,” through the other input 78 which are communicated
`to the microprocessor 36 by the local microprocessor 68 or
`directly, e.g., using a game port. The user object 76 can take
`many forms, including a joystick, mouse, trackball, steering
`wheel, medical instrument, representation of a body part,
`gamepad controller, etc., as described in U.S. Pat. No.
`5,734,373 and application Ser. Nos. 08/664,086 and 08/881,
`691, all incorporated by reference herein.
`[0047] If the pointer on the display device 64 is at a
`position (or time) that correlates to a desired force feedback
`
`to the user 52, or an event occurs that dictates that force
`feedback should be output, the microprocessor 36 sends a
`force feedback command to the local microprocessor 68
`over the serial port connection 88. The local microprocessor
`68 parses this force feedback command and sends signals to
`the actuator interface 80 which causes the actuator 84 to
`create forces F on user object 76, which are experienced by
`the user 52 as indicated at 60. The safety switch 82,
`sometimes referred to as a “deadman switch”, blocks the
`signal from the actuator interface 80 if, for example, the user
`52 is no longer grasping the object 76. In this way, the user
`52 can interact with the client machine 46 in a visual,
`auditory, and tactile fashion.
`[0048] For example, when using the local microprocessor
`68 to offload computational burden from the host computer,
`the host can send high level commands to the local micro
`processor 68. The local microprocessor can parse or inter
`pret the commands and implement a local force routine that
`is stored in local memory 71. Such a force routine might
`instruct the microprocessor 68 to read sensor positions,
`determine a force based on the sensor positions and com
`mand the actuators 84 to output the force, all in a local
`control loop independent from the host computer (the micro
`processor 68 would also preferably relay the sensor posi
`tions to the host computer). Different force routines can be
`provided to command