US 6,557,041 B2
`(io) Patent No.:
`a2) United States Patent
`Mallart
`(45) Date of Patent:
`*Apr. 29, 2003
`
`
`US006557041B2
`
`(54) REAL TIME VIDEO GAME USES
`EMULATION OF STREAMING OVER THE
`INTERNET IN A BROADCAST EVENT
`
`x
`(75)
`
`.
`.
`.
`Inventor: Raoul Mallart, Mountain View, CA
`(
`)
`.
`.
`.
`+s
`we
`.
`(73) Assignee: Koninklijke Philips Electronics N.V.,
`Eindhoven (NL)
`
`6,006,254 A * 12/1999 Waters et al... 709/205
`6,055,563 A *
`4/2000 Endo etal.
`...
`... 709/203
`6,185,602 B1 *
`2/2001 Bayrakeri
`.......0..0.00. 709/204
`6,204,862 B1 *
`3/2001 Barstow et al.
`..
`eee 345/473
`6,208,357 Bl *
`3/2001 Koga etal.
`......
`wee 345/473
`6.215.498 Bl *
`4/2001 Filo etal. .....
`we 345/419
`6,219,045 B1 *
`4/2001 Leahyetal. ..
`we. 345/331
`
`6,285,379 B1 * 11/2001 Gallery «00... 345/473
`.
`«345/419
`6,331,851 B1 * 12/2001 Suzukiet al.
`2/2002 Geveretal. cues. 345/473
`6,351,267 Bl *
`
`
`
`(*) Notice:
`
`This patent issued on a continued pros-
`ecution application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`patent
`term provisions of 35 U.S.C.
`154(a)(2).
`
`Subject to anydisclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`OTHER PUBLICATIONS
`Katkere A Et Al: “VRML-—Based WWW Interface To MPI
`
`Video” Dec. 14-15, 1995, p. 25-31 XP000791035 the whole
`document.
`Barrus J.W; Waters R.C ; Anderson D.B. : “locales: sup-
`porting large multiuser virtual environments” vol. 16,No. 6
`Nov. 1996 pp. 50-57 and 51-53 XP002125829.
`<http://www.virtualive.com> of Orad Hi-lec Systems,
`Ltd.;.
`
`(21) Appl. No.: 09/149,950
`(22) Filed:
`Sep. 9, 1998
`(65)
`Prior Publication Data
`US 2003/0037156 Al Feb. 20, 2003
`
`(List continued on next page.)
`Primary Examiner—Zarni Maung
`Assistant Examiner—Abdullahi E. Salad
`(74) Attorney, Agent, or Firm—Gwenaelle Kasriel
`
`Related U.S. Application Data
`
`(57)
`
`ABSTRACT
`,
`:
`In a broadcast application on a client-server network the
`streaming is emulated of animation data over the Internet to
`(63) Continuation-in-part of application No. 09/138,782,filed on
`a large number of clients. The animation is considered a
`Aug. 24, 1998.
`sequence of states. State information is sent to the clients
`51) Int.Cl’ CO6F 15/16
`instead of the graphics data itself. The clients generate the
`° USs. Cl
`709/231: 709/205; 345an
`animation data itself under control of the state information.
`(52) Ones >
`mete
`The scrver and clicnts communicate using a shared object
`(58) Field of Search oe 709/204, 205,
`protocol. Thus, streaming is accomplished as well as a
`709/237, 231, 210; 345/473
`broadcast without running into severe network bandwidth
`:
`(56)
`.
`.
`.
`References Cited
`problems. This approachis used to mapareallife event, e.g.,
`U.S. PATENT DOCUMENTS
`a motor race, onto a virtual environmentin orderto let the
`user participate in a virtual race against
`the real
`life
`professionals, the dynamicsof the virtual environmentbeing
`determined by the state changes sent to the user.
`.
`.
`
`8/1997 Durward et al.
`........... 395/329
`5,659,691 A
`5/1998 Duttonet al......
`-» 345/473
`5,754,190 A *
`
`. 709/231
`9/1999 Kalra et al.
`.......
`5,953,506 A *
`
`9/1999 Matsuietal. ..
`.. 345/329
`5,956,028 A *
`5,999,944 A * 12/1999 Lipkin 0... ee 345/502
`
`9 Claims, 6 Drawing Sheets
`
`612
`
`
`612
`608
`
`608
`610
` 610\
`
`\
`\
`
`
`00
`NETWORK
`LIC
`
`
`
`
`
` SERVER STATE
`INFO
`
`608
`
`610
`
`
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0001
`EXHIBIT 1028 - PAGE 0001
`
`

`

`US 6,557,041 B2
`Page 2
`
`OTHER PUBLICATIONS
`
`“Key Concepts”, Mar. 5, 1996, at:<http://sgi.felk.cvut.-
`cz/~holecek/VRML/concepts.html>;.
`“Internetwork Infrastructure Requirements for Virtual Envi-
`ronments”, D.P. Brutzmanet al., Jan. 23, 1996, at: <http://
`www.stl.ops.navy.mil/~brutzman/vrml/vrml_95.html>;.
`“Overview of the MPEG—4 Standard”,
`ISOMEC JTC1/
`$C29/WG11 N2323 ed. Rob Koenen, Jul. 1998, at <http://
`drogo.cselt.stet.it/mpeg/ standards/mpeg—4/mpeg—4.htm>;.
`
`“Design of the Interactive Sharing Transfer Protocol”, at
`<http://www.hir.com/opencom/istp.html>;.
`
`“Dead Reckoning in DIS”, at <http://www.ist.ucf.edu/lab-
`sproj/projects/drhtml >>;.
`
`“DIS—Protocol”, at <http/Awww.cg.tuwien.ac.at/~faisst/LS-
`DMVE/dis.html >>.
`
`* cited by examiner
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0002
`EXHIBIT 1028 - PAGE 0002
`
`

`

`U.S. Patent
`
`Apr. 29, 2003
`
`Sheet 1 of 6
`
`US 6,557,041 B2
`
`116.
`
`118
`
`~eeeeeeeeei.‘Oo
`meerwereeKeerpyrterecreereeei'oO\'a1a|Bo!25'!XOI.'cls
` toc'mT!cotG1'aoac=>!=Mua18o2—Li¢1aaSor
`
`2tooeIt'7aI1oN\'op)1're1IooaKEt=!'a4t—Oi'—l
`4'4i]’”
`t‘=''IiorIS1ud
`<r“—iS:o™oO='i,iwrt-1!be.1Oo'i'ioO.
`tii1—,icS''aLu
`io.'Ouii&
`
`
`c5roeoeenapanscsce
` coLioS“
`
`
`td
`
`202°"
`
`“204
`
`FIG. 2
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0003
`EXHIBIT 1028 - PAGE 0003
`
`
`
`

`

`U.S. Patent
`
`Apr. 29, 2003
`
`
`
`JOYNOSTSNNVHO
`
`TANNVHO
`
`eend
`
`OE
`
`WIKIS|WY3YLS
`
`JOUNOS
`
`Sheet 2 of 6
`
`US 6,557,041 B2
`
`
`
`44000404300040
`
`YIGOINI
`
`YACOONS
`
`
`
`MOLNAN9'one
`
`YSAMAS|
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0004
`EXHIBIT 1028 - PAGE 0004
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 29, 2003
`
`Sheet 3 of 6
`
`US 6,557,041 B2
`
`WVSYLS
`
`LO
`
`HIAVIdS4IG“ELNINO
`
`
`
`(IN3I19b-93dWN)
`
`=
`
`TANNVHO
`
`4400040
`
`TANNVHO
`
`Y4AGQON3
`
`JOUNOS
`
`WGOONS
`
`WVSYLS
`
`NI
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0005
`EXHIBIT 1028 - PAGE 0005
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr.29, 2003
`
`US 6,557,041 B2
`
`Sheet 4 of 6
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0006
`EXHIBIT 1028 - PAGE 0006
`
`

`

`U.S. Patent
`
`Apr.29, 2003
`
`Sheet 5 of 6
`
`US 6,557,041 B2
`
`
`
`
`
`NETWORK
`
`FIG. 7
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0007
`EXHIBIT 1028 - PAGE 0007
`
`

`

`U.S. Patent
`
`Apr.29,
`
`2 F
`
`IG. 8
`
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0008
`
`

`

`US 6,557,041 B2
`
`1
`REAL TIME VIDEO GAME USES
`EMULATION OF STREAMING OVER THE
`INTERNET IN A BROADCAST EVENT
`
`CROSS-REFERENCE
`
`This is a Continuation-in-part of application Ser. No.
`09/138,782 filed Aug. 24, 1998 for Raoul Mallart and Atul
`Sinha, “EMULATION OF STREAMING OVER THE
`INTERNET IN A BROADCASTAPPLICATION”.
`
`FIELD OF THE INVENTION
`
`The invention relates to streaming multimedia files via a
`network. The invention relates in particular to enabling the
`emulation of streaming graphics or video animation over the
`Internet within a broadcast context. The inventionrelates in
`
`particular to enabling an end-user to interact with the
`environment created through the graphics or video anima-
`tion.
`
`BACKGROUND ART
`
`The term “streaming” refers to transferring data from a
`server to a client so that it can be processed as a steady and
`continuousstream at the receiving end. Streaming technolo-
`gies are becomingincreasingly important with the growth of
`the Internet because most users do not have fast enough
`access to download large multimedia files comprising, e.g.,
`graphics animation, audio, video, or a combination thereof,
`elc. Streaming, however, enables the client’s browser or
`plug-in to start processing the data before the entire file has
`been received. For streaming to work,the client side receiv-
`ing the file must be able to collect the data and send it as a
`steady stream to the application that is processing the data.
`This means that if the client receives the data faster than
`
`required, the excess data needs to be buffered. If the data
`does not arrive in time, on the other hand, the presentation
`of the data will not be smooth.
`
`The term “file” is used herein to indicate an entity of
`related data items available to a data processing and capable
`of being processed as an entity. Within the context of the
`invention, the term “file”? may refer to data generated in
`real-time as well as data retrieved from storage.
`Among the technologies that are currently available or
`under development for the communication of graphics data
`via the Internet are VRML 97 and MPEG-4. VRML 97
`stands for “Virtual Reality Modeling Language”, and is an
`International Standard (ISOMEC 14772)
`file format
`for
`describing interactive 3D multimedia content on the Inter-
`net. MPEG-4 is an ISO/IEC standard being developed by
`MPEG (Moving Picture Experts Group). In both standards,
`the graphical contentis structured in a so-called scene graph.
`A scene graph is a family tree of coordinate systems and
`shapes,
`that collectively describe a graphics world. The
`top-most item in the scene family tree is the world coordi-
`nate system. The world coordinate system acts as the parent
`for one or more child coordinate systems and shapes. Those
`child coordinate systemsare, in turn, parents to further child
`coordinate systems and shapes, and so on.
`VRMLis a file format for describing objects. VRML
`defines a set of objects useful for doing 3D graphics,
`multi-media, and interactive object/world building. These
`objects are called nodes, and contain elemental data which
`is stored in fields and events. Typically, the scene graph
`comprises structural nodes, leaf nodes, interpolation nodes
`and sensor nodes. The structural nodes define the spatial
`relationship of objects within a scene. The leaf nodes define
`
`10
`
`15
`
`30
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`the physical appearance of the objects. The interpolation
`nodes define animations. The sensor nodes define user
`interaction for particular user input modalities. VRML does
`not directly support streaming of data from a server into a
`client. Facilities such as synchronization between streams
`and time stamping thatare essential in streaming do not exist
`in VRML. However, VRML has a mechanism that allows
`external programs to interact with VRMLclients. This has
`been used in sports applications to load animation data into
`the client. See, for example, “VirtuaLive Soccer” of Orad
`Hi-Tec Systems, [td at <http:/Avww.virtualive.com>. This
`web document discusses a process for producing realistic,
`animated,
`three-dimensional graphic clips that simulate
`actual soccer match highlights for being sentvia the Internet.
`The system generates content that complements television
`sports coverage with multimedia-rich Web pagesin near real
`time. In this example, the process works in twosteps. First
`the graphics models of the stadium andofthe soccer players
`are downloaded along with an external program,in this case
`a Java Applet. ‘he user can then interact with the external
`program to request a particular animation. The data for this
`animation is then downloaded into the clicnt and interacted
`
`with by the user: the user can view scenes of the match from
`different points of view in the animation and in slow motion
`if desired. In termsof node type, this process first downloads
`the structural and leaf nodes, and thereuponthe interpolation
`nodes. By changing the set of interpolation nodes,
`it
`is
`possible to run a different animation sequence. The process
`used in this example is somewhat equivalentto a single step
`process in which the user can choose the complete VRML
`file that contains all the models (structural nodes) and all the
`animation data (interpolator nodes). This approach leads to
`long download times before any content can be played on
`the client. This is experienced as a frustrating experience,
`especially if compared to TV broadcast where content is
`available instantly.
`The other technology introduced above, MPEG-4, defines
`a binary description format for scenes (BIFS)that has a wide
`overlap with VRML 97. MPEG-4, on the other hand, has
`been designed to support streaming of graphics as well as for
`video. MPEG-4 defines two server/client protocols for
`updating and animating scenes: BIFS-Update and BIFS-
`Anim. Some of the advantages of MPEG-4 over VRMLare
`the coding of the scene description and of the animation data
`as well as the built-in streaming capability. The user does not
`have to wait for the complete download of the animation
`data. For example, in the soccer match broadcast application
`mentioned earlier the animation an start as soon as the
`models of the players and the stadium are downloaded.
`MPEG-4 further has the advantage that it more efficient
`owingto its BIFS transport protocol that uses a compressed
`binary format.
`Within the context of streaming, the knowntechnologies
`mentioned above have several limitations with regard to
`bandwidth usage, packet-loss concealment or recovery and
`multi-user interactivity, especially in a broadcast to large
`numbers ofclients.
`
`As to bandwidth, the complete animation is generated at
`the server. This results in a large amount of data that needs
`to be transported over the network, e.g., the Internet, con-
`necting the client to the server. For example, in the soccer
`broadcast application mentioned above, the 22 soccer play-
`ers need to be animated. Each animation data point per
`individual player comprises a position in 3D space anda set
`of, say, 15 joint rotations to model the player’s posture. This
`represents 63 floating-point values.If it is assumed that the
`animation updaterate is 15 data points per seconds,a bil-rale
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0009
`EXHIBIT 1028 - PAGE 0009
`
`

`

`US 6,557,041 B2
`
`10
`
`15
`
`for example, a
`is,
`the virtual soccer match, an object
`graphics representation of one of the soccer players. The
`server receives a streamed video file and monitors the
`
`35
`
`3
`4
`gencrate the multimedia file under control of the respective
`of 665 Kbpsis required. Thisbit-rate can be reduced through
`state information. In a broadcast for animation, the invention
`compression. Typically, using BIFS reducesthe bit-rate by
`relates to a method of supplying data via a network for
`a factor of 20, giving a bit-rate of about 33 Kbps. However,
`enabling displaying graphics animation. Respective state
`this numberhas not taken into account overhead required for
`information is supplied over the network descriptive of
`the Internet protocols (RTP, UDPand IP) and for additional
`successive respective states of the animation. The respective
`data types, such as audio. However, typical modems cur-
`state information is received via the network. The receiving
`rently commercially available on the consumer market have
`station is enabled to generate the animation under control of
`a capacity of 28.8 Kbps or 33.6 Kpbs.
`It
`is clear that
`the respective state information upon receipt. More
`streaming animation causes a problem at the end user due to
`particularly, the invention provides a method of enabling a
`bandwidth limitations. In the case of a broadcast to a large
`user to navigate in a continuously evolving electronic virtual
`numberof clients, say 100,000 clients, the data stream will
`environment. The method comprises: providing to the user
`need to be duplicated at several routers. A router on the
`a world model of the environment; sending state changes of
`Internet determines the next network point to which a packet
`the world model representative of the evolving; enabling the
`should be forwarded on its way toward its final destination.
`user to provide input for control of a position of an object
`The router decides which way to send each information
`relative to the virtual environment; and creating the envi-
`packet based onits current understanding of the state of the
`ronment from the world model and in response to the state
`networks it
`is connected to. A router is located at any
`changes and the user input.
`juncture of networks or gateway, including each Internet
`In the invention the multimediafile (animation, video or
`point-of-presence. It is clear that the broadcast could lead to
`audio file) is described as a succession ofstates. It is this
`an unmanageable data explosion across the Internet. To
`state information that gets transmitted to the clients rather
`prevent that from happening, the actual bandwidth needs to
`than the animation data itself The term “emulating” there-
`be limited to much lower than 28.8 Kbps.
`fore emphasizes that the information communicated to the
`As to packct loss conccalment, VRML-based systems
`client need not be streamed. The client generates the data for
`utilize reliable protocols (TCP). Packet losses are not an
`play-out locally and based on the state information received.
`issue here. In the case of MPEG-4, BIFS uses RTP/UDP/IP.
`Accordingly, the user perceives a continuous and steady
`A packet loss recovery mechanism is therefore required. In
`stream of data during play-out as if the data were streamed
`a point-to-point application, re-transmission of lost packets
`over the network (under optimal conditions).
`can be considered. In a broadcast situation, however, this is
`In a preferred embodiment, a shared-object protocol is
`much more complex. In both cases, however, MPEGreli-
`30
`used to accomplish the emulation. Both a server andaclient
`ability requires either higher bandwidth usage (redundancy)
`have copies of a collection of objects. An object is a data
`or higher latency (retransmission).
`structure that holds state information. Within the context of
`As to multi-user interactivity, both VRML and MPEG-4
`are essentially based on a server-client communication. No
`provisions exist to enable communication among multiple
`clients.
`
`For more information on VRMLsee, for example, “Key
`Concepts”, Mar. 5, 1996, at: <http://sgi.felk.cvut.cz/
`~holecek/VRML/concepts.html>, or “Internetwork Infra-
`structure Requirements for Virtual Environments”, D. P.
`Brutzman et al., Jan. 23, 1996, publicly available at: <http://
`www.stl.ops.navy.mil/~brutzman/vrml/vrml,, 95.html>.
`For more information on MPEG-4 see,
`for example,
`“Overview of the MPEG-4 Standard”, ISOAEC JTC1/SC29/
`WG11 N2323 ed. Rob Koenen, July 1998, publicly available
`at <http://drogo.cselt.stet.it/mpeg/standards/mpeg-4/mpeg-
`4.htm>.
`
`OBJECT OF THE INVENTION
`
`It is therefore an object of the invention to provide a
`technology that enables a client to process multimedia data
`as if it were a steady and continuousstream. It is another
`object to enable the continuous processing at a large number
`of clients in a broadcast over the Internet. It is noted that the
`
`problemsidentified above becomerather acute in a broad-
`cast application. It is another object to use this technology
`for creating an interactive software application that enables
`the user to navigate in a continuously evolving electronic
`virtual environment.
`
`SUMMARYOF THE INVENTION
`
`To this end, the invention provides a method of emulating
`streaming a multimedia file via a network to a receiving
`station connected to the network. Respective state informa-
`tion descriptive of respective states of the file is supplied.
`The receiving station is enabled to receive the respective
`stale information via the network and is enabled to locally
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`objects. It is noted that MPEG-4 enables the creation of
`video objects that are processed as an entity. If the server
`changes the state of this object, the shared object protocol
`causes the copy at the client to change accordingly. This is
`explained in more detail with reference to the drawings.
`This state information is at a higher level of abstraction
`than the animation data itself. For example, in the soccer
`match broadcast application mentioned above,
`the state
`information comprises the current positionsof the 22 players
`in the field and parameters specifying their current action
`(e.g., “running”, “jumping”, etc.). The use of higher level
`information has several advantages, in particular in a broad-
`cast application where animation is streamed overthe Inter-
`net to a large audience. The content of the state information
`as communicated over the Internet is very compact, thus
`requiring lower bandwidth than in case the animation data
`itself is streamed. The animation is generated locally from a
`few parameters. In addition, the update rate of animation
`data points is lower because the state of the animation
`changesat a slower rate than the animation data itself. This
`contributes to further lowering bandwidth requirements.
`Furthermore, the invention provides enhanced possibilities
`for packet loss recovery or concealment and for network
`latency jitter masking. It is easy to interpolate or extrapolate
`between states and to implement dead reckoning concepts.
`Userinteraction with the animation is more easily program-
`mable because of this higher level of abstraction. Another
`advantage is that multi-user interaction is feasible if clients
`are enabled to share state information. Still another advan-
`
`tage is the fact that clients are enabled to convert the state
`information into animation based on their individual pro-
`cessing power that might differ from client to client. The
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0010
`EXHIBIT 1028 - PAGE 0010
`
`

`

`US 6,557,041 B2
`
`5
`resources available at the clicnt may be different per client
`or groups ofclients.
`Within the context of the invention, reference is made to
`US. patent application Ser. No. 09/053,448 (PHA 23,383) of
`same Assignee, titled “Group-wise video conferencing uses
`3D-graphics model of broadcast event” and incorporated
`herein by reference. This document addresses a TV broad-
`cast service to multiple geographically distributed end users.
`The broadcast service is integrated with a conferencing
`mode. Upona certain cvent in the broadcast, specific groups
`of end users are switched to a conference mode under
`software control so that the group is enabled to discuss the
`event. The conference mode is enhanced by a 3D graphics
`model of the video representation of the event
`that
`is
`downloaded to the groups. The end users are capable of
`interacting with the model
`to discuss alternatives to the
`event.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The invention is explained by way of example and with
`reference to the accompanying drawings, wherein:
`FIG. 1 is a diagram of a VRMLclient-server system;
`TIG. 2 is a diagram of an MPLG-4 client-server system;
`FIGS. 3-6 are diagrams of systems used in the invention;
`FIGS. 7-8 are diagrams illustrating the video game
`Throughoutthe figures, same reference numerals indicate
`similar or corresponding features.
`PREFERRED EMBODIMENTS
`
`FIG. 1 is a block diagram of a client-server system 100
`based on VRML. System 100 comprises a server 102
`coupled with a client 104 via a communication channel 106,
`here the Internet. System 100 may comprise moreclients but
`these are not shown in order to not obscure the drawing.
`Server 102 comprises a source encoder 108 and a channel
`encoder 110. Client 104 comprises a channel decoder 112
`and a source decoder 114. Source encoder 108 is considered
`
`a content generation tool. For example, it can be a toolthat
`generates the VRML animation data from motion capture
`devices (not shown) operating on video. Channel encoder
`110 is a sub-system that takes as input the VRMT, animation
`generated at source encoder 108 and transforms it into a
`form that can be transported over the Internet. The VRML
`animation data is stored in a file. The transport of this file
`uses a standardfile transport protocol. At client 104, channel
`decoder is contained in an external program 116. It gets the
`animation data from the downloaded file and sendsit to a
`VRMLplayer 118 that performs the source decoder func-
`tion. The source decoder function is essentially a manage-
`ment of the scene graph. This server-client communication
`procedure is not a streaming solution. The specification of
`VRMLdocsnot consider streaming a requirement. Facilitics
`such as synchronization between streams and time stamping,
`both essential to streaming, do not exist in VRML.
`FIG. 2 is a block diagramof a client-server system 200
`based on MPEG-4. System 200 has a server 202 coupled to
`a client 204 via a communication channel 206. Server 202
`has a source encoder 208 and a channel encoder 210. Client
`204 has a channel decoder 212 and a source decoder 214. As
`
`mentioned above, MPEG-4 has been designed to support
`streaming. Among other things, MPEG-4 has defined a
`binary description format for scenes (BIFS) that has a wide
`overlap with VRML 97. Furthermore, MPEG-4 defines two
`server/client protocols for updating and animating scenes,
`namely BIFS-Update and BIFS-Anim. Advantages of
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`MPEG-4 over VRMLwithin the context of streaming is the
`coding of the scene description and of the animation data as
`well as the built-in streaming capability. Source encoder 208
`is, similarly to encoder 108, a content generation tool.
`Channel encoder 210 is different from channel encoder 110.
`It generates a bit-stream in BIFS and BIFS-Anim formats.
`This bit-stream contains the graphics models of the players
`and the stadium (in the soccer match animation) as well as
`the animation data.
`
`However, both systems 100 and 200 have several serious
`drawbacks when used in an environment for broadcasting
`animation to a large number of clients, say, 100-100,000
`clients. ‘The limitations relate to network bandwidth usage,
`packet
`loss concealment and multi-user interactivity as
`already mentioned above.
`A preferred embodiment of the invention provides a
`solution to these problems by emulating the streaming
`utilizing a communication protocol that supports the sharing
`of objects by an object owner and an object viewer (or
`listener). A shared abject is a data structure that holds state
`information. The set of shared objects that defines the entire
`state is called the world model. The clients and the server
`have their own copy of the world model. For example, an
`object within-the context of a soccer match representation is
`the representation of a soccer player. The object’s state
`information is then, for example, the position of the soccer
`player in 3D space or an action state such as “running” or
`“jumping” or “sliding” or “lying on the ground, apparently
`hurt, but has track record of comedian”. Each shared object
`is owned by a particular party, e.g., the server. The owner
`can change the state information contained in the object.
`Whenthis occurs, the protocol automatically synchronizes
`the state information across the network. Such a protocol is
`referred to herein below as a shared-object support protocol.
`The protocol ensuresthat all the world model copies remain
`consistentas the state of the world model evolves. Examples
`of protocols that can be used for this purpose are DIS
`(Distributed Interactive Simulation) and ISTP (Interactive
`Sharing Transfer Protocol). An underlying idea of the inven-
`tion is to describe the animation as a succession of states. For
`
`example, in the soccer software application, the animationis
`described as a succession of player positions onthe field and
`action states of the players. The state at a given momentis
`represented by the world modcl. As time passcs, the state
`evolves and the protocol synchronizes the state of the world
`model across the network. This can also be explained in
`terms of shared objects. These objects hold the state infor-
`mation that describes the game at a given time instant.
`Updates of the state information for each object result in the
`generation of messages that are being sent across the net-
`work to the clients.
`
`FIG. 3 is a block diagram of a system 300 in the invention.
`System 300 comprises a server 302 coupled to a client 304
`via a network 306. Server 302 comprises a source encoder
`308 and a channel encoder 310. Client 304 comprises a
`channel decoder 312 and a source decoder 314. Server 302
`
`has a copy 316 of a world model and client 304 has a copy
`318 of the world model. Data is streamed to source encoder
`308 at an input 320. Source encoder 308 generates the
`required state information based on the input received and
`updates the state of the objects in world model copy 316 as
`the streaming process continues. This type of technology is
`used, for example, by the VirtuaLive Soccer system men-
`tioned above. Channel encoder 310 monitors world model
`
`copy 316 and encodes the state changes of the shared
`objects. The encoded state changes are sentto client 304 via
`network 306. Channel decoder receives the slate changes
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1028 - PAGE 0011
`EXHIBIT 1028 - PAGE 0011
`
`

`

`US 6,557,041 B2
`
`7
`and updates local world model copy 318. Source decoder
`314 performs twotasks. First, it generates the animation
`based on the state information received. Secondly, source
`decoder 314 manages the scene graph according to the
`animation. Source decoder 314 is nowan intelligent com-
`ponent: it performs animation computation and, in addition,
`it is capable of performing other tasks such as state inter-
`polation or extrapolation to conceal packet losses or network
`latency jitter.
`Within this context reference is made to U.S. patent
`application Ser. No. 08/722,414 (PHA 23,155) of same
`Assignee,
`titled “Multi-player video game with local
`updates mitigates latency effects”, herein incorporated by
`reference. This reference discusses a system wherein mul-
`tiple users share a virtual environmentthroughaninteractive
`software application. State changes of a specific user are
`transmitted to one or more other users dependent on respec-
`tive relative distances in the virtual environment between the
`specific user and each respective one of the other users. This
`conditional transmission reduces messagetraffic and allows
`the virtual environment to scale practically indefinitely.
`Also, reference is made to U.S. patent application Ser. No.
`08/722,413 (PHA 23,156) of same Assignee,titled “Latency
`effect
`in multi-player video game reduced by surrogate
`agent” and incorporated herein by reference. This document
`relates to a data processing system that processes an inter-
`active software application for a competition between two or
`more users. The system comprises user-interface machines
`for actuation by respective ones of the uscrs. The machines
`are interconnected via a network. In order to effectively
`eliminate latency, a user is represented at the other users’
`machinesas an agent whosereaction to an action ofthe other
`user is governed by a rule base stored in the system.
`Further reference is made to U.S. patent application Ser.
`No. 08/994,827 (PHA 23,319) of same Assignee and incor-
`porated herein by reference, titled “Diversion agent uses
`cinematographic techniques to mask latency”. This docu-
`ment relates to a software agent as a functional part of a
`user-interactive software application running on a data pro-
`cessing system. The agent creates a user-perceptible effect in
`order to mask latency present in deliveryof data to the user.
`The agent creates the effect employing cinematographic
`techniques.
`It is noted that copies 316 and 318 of the world model
`necd not be identical, c.g., in appearance when rendered, as
`long as an object in one copy of the world model and another
`object in another copy of the world model are being treated
`as shared in the sense that they share state changes. The
`feasibility and extent of non-identity is application-
`dependent. For example, if one client’s user likes to repre-
`sent the soccer players as, say, penguins, and anotherclient’s
`user prefers the representation of say, ballet dancers,
`the
`representation at both clients are being kept consistent
`throughout the system by meansofthe shared state changes.
`As another example, client 304 may enable the user to input
`additional state information to control the rendering of the
`world model at play-out. For example, the user may select
`a particular point of view when watching the VirtuaLive
`soccer match. This state information is not, and need not, be
`present in server 302. It is noted that the rendering of the
`viewpoint based on state information and the world modelis
`muchless complicated and requires fewer resources than if
`the image were actually streamed into client 304 as a bitmap
`with depth information. Accordingly,
`in addition to the
`advantages of the invention mentioned earlier, the invention
`facilitates user-interactivity.
`The configuration of system 300 assumesthat client 304
`is capable of running a software application and has a
`
`8
`sufficicntly powerful CPU and a sufficiently large storage.
`Some clients may not have these capabilities onboard. It is
`therefore desirable to consider lower-end terminals, also
`referred to as “thin clients”. Such terminals could be, for
`example, low-profile MPEG-4 terminals that accept a BIFS
`stream as input but may not be powerful enough. This is
`explained with reference to FIG. 4 that is a block diagram of
`a system 400 according to the invention. System 400 com-
`prises a server 302 that communicates with client 204 via a
`translator station 406. The configuration of server 302 and
`client 204 has been discussed above. Translator station 406
`
`maintains a local copy of the world model. This world model
`is updated by the messages from server 302 so that the
`model represents the current state. Based on this state
`information, translator station 406 computes the animation.
`The animation data is encoded in BIFS-Anim format and
`transmitted to MPEG-4 client 204. Server 302 is similar to
`the one in system 300. Translator station 406 is a modulethat
`performs a conversion between messages transmitted under
`the s