`
`
` Nathaniel Polish, Ph.D.
`Director of Technology, Instant Video Technologies
`January 1996
`
`Burstware ™ is a family of computer network communication protocols embodied in
`software layers and useful in applications involving multimedia objects. All
`Burstware ™ protocols use substantial amounts of memory on the client system. In
`most cases, "substantial" means at least a megabyte or more of memory, compared
`to typical network packet sizes of 1500 bytes or cell sizes of 50 bytes. All Burstware
`™ protocols involve a close relationship between client and server. Instead of an
`arrangement in which the client requests what it needs from the server, and hopes
`that the server has the resources to comply in a timely manner, Burstware ™
`protocols create a client/server relationship in which the server is aware of each
`client's needs. The server fills each client's needs in a manner that obtains the most
`from server and network resources. The environment which best takes advantage
`of this kind of client/server relationship is one in which the data is video and/or
`audio, and the network can deliver bandwidths which are greater than necessary for
`real-‐time transmission of the media.
`
`There are many uses of memory in current network communication systems. The
`major use of memory on systems such as TCP/IP networks is in the transfer of data
`packets. On the receiver (client) side, packets must be received completely into
`memory, decoded, checksummed (checked for errors), and delivered to the
`addressee. On the transmitter (server) side, packets must be held in memory until
`receipt of the transmission has been confirmed; in the event of a transmission error,
`they may have to be re-‐transmitted. Since packet sizes are relatively small (1500
`bytes in most TCP/IP implementations,) the amount of memory on a client protocol
`stack is fairly small.
` On the transmission side, the server's need for memory increases as the network
`bandwidth increases. This can be seen by examining two important characteristics
`of a network: bandwidth and latency. Bandwidth is the number of bits delivered per
`second. Latency is the amount of time that a bit spends in the pipeline between the
`transmitter and receiver. The speed of light is one component of latency; however,
`it is common on packet-‐switched networks (such as the Internet) for a packet to be
`Instant Video Technologies, Inc.
` CONFIDENTIAL
`Page 1
`
`
`
`1.0 What is "Burstware ™"?
`
`1.1 Memory
`
`PAGE 1 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
`received and re-‐transmitted several times before reaching its addressee. Each
`transmission adds processing delays that contribute to latency.
` On a given network connection, the number of bits in the pipeline is given by the
`bandwidth times the latency. Latencies tend to be fixed, or at best, shrink slowly
`with improvements in technology. Local area networks (LANs) have latencies from
`1ms to several tens of milliseconds. Wide area networks (WANs) have latencies
`from 100ms to over 1000ms. Bandwidths however, are growing fast. At 9600 bits
`per second on a network with 200ms, latency would be less than 2000 bits in the
`pipeline at any one time. At 155mbs with 100ms, latency would be 15 megabits in
`the pipeline at any one time. This means that the server must have at least 15
`megabits of memory if it needs to, in the event of errors, support re-‐transmission,
`and still continuously transmit data. Usually, lack of sufficient memory results in the
`server slowing its transmission rate.
` Use of network file systems for supporting the play of multimedia files adds another
`set of requirements. Multimedia files (audio and video) are naturally played
`(consumed) at certain rates. These rates may be constant (constant bit rate, or CBR)
`or variable (VBR), as in the case of MPEG compressed video. In any event, if the
`network file system can not guarantee near instantaneous delivery at the
`consumption rate, then the multimedia file playback will fail. If the latency and
`bandwidth parameters are constant, then as long as there are no other delays,
`playback will work. However, as we will see in Section 3, both the latency and the
`bandwidth available on a network will vary moment-‐to-‐moment as a consequence
`of network traffic and errors. Burstware ™ protocols create a environment within
`the network that enables the network to deal with latency and bandwidth
`fluctuations in such a way, that multimedia files can be reliably and instantaneously
`delivered.
`
`In general, computer networks are effective at delivering bulk data in correct order,
`with no errors. The average bandwidth over several seconds can usually be
`assured, as can the average latency. However, short time scale values for bandwidth
`and latency usually cannot be guaranteed. In this case, "short time scale" is a
`relative term that depends on many things; nevertheless, in most current situations,
`short time scales are on the order of tenths of a second. In a network environment
`in which the average bandwidth is faster than the real-‐time demands of multimedia
`file playback, a burst approach can be used to send multimedia file data.
` A metaphor which may be useful in understanding multimedia file transmission is
`this: Each client has a large bucket into which water (data) is poured. The water is
`delivered at wide intervals, but when it is delivered, a
`Instant Video Technologies, Inc.
` CONFIDENTIAL
`
`1.2 Buckets with Faucets and Open Tops
`
`Page 2
`
`PAGE 2 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
`1.3 Client/Server Management
`
`large amount is delivered. The server keeps track of how full each of its client's
`buckets is moment-‐to-‐moment, as well as how long it will take to reach each bucket
`with a new supply of water. The clients are draining the buckets through a faucet at
`whichever rate suits them. It is the server's job to make sure that all of the buckets
`are kept as full as possible — to keep track of changes in the time it will take to get
`to each bucket, and rates at which each client is using its water supply.
` The server schedules deliveries to client buckets and clients consume water. The
`trick is to choose the size of the buckets, and the delivery schedule such that the taps
`never run dry. Typical client/server relationships can be described as "fill the
`buckets as fast as possible," or "fill the buckets only as the water is consumed from
`the tap." A more productive network relationship would be for the server to give
`preference to buckets that are running close to empty over ones that are more
`nearly full.
`
`The Burstware ™ protocols take the approach that the network system as a whole,
`not just individual client/server relationships, needs to be managed. The typical
`assumption in network computing is that the clients are greedy. This is certainly the
`case, for example, with the FTP protocol. In this protocol, the client program takes
`as much as it can, as fast as it can, from the server. It is left to both the server and
`the network operating environment to deliver some level of service that is
`"equitable" to the rest of the network. This works fine for situations such as FTP,
`where the semantics of the user's request is: "get me this entire file as soon as you
`can."
` In multimedia file playback environments, the semantics of the client's requests are
`different from the traditional file transfer semantics. Further, the consequences of
`applying greedy scheduling to a network supplying multimedia clients does not
`result in the best service to the most clients. This can best be seen by considering
`that a client who is about to run out of content material should receive packets
`sooner than clients who have extensive buffers. Since clients cannot directly
`cooperate with each other, the server needs to maintain information about their
`needs and deliver service based on need, rather than on who has the bigger pipe, or
`who was first at the trough.
` When using Burstware ™ protocols, the clients regularly inform the server as to
`their status, the server regularly probes the network to keep track of the prevailing
`conditions on the network, and the server is constantly updating its schedule of
`transmission activity to properly serve the clients.
`
`Instant Video Technologies, Inc.
`
` CONFIDENTIAL
`
`Page 3
`
`PAGE 3 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
`2.0 How it Works
`
`Typical digital video flow is frame by frame at 30 frames per second. This
`characterizes 601-‐type video streams. This is known as CBR because each frame
`holds the same amount of data, and each frame is transmitted at a constant,
`predictable rate. (See Figure 1 below.)
`
`Video Frames
`
`
`Figure 1. CBR video frames. Each frame contains the same amount of data, and
`frames are transmitted at a constant rate of frames per second.
` This sort of coding takes no advantage of redundancy in the frames; successive
`frames are rarely very different from each other. Compression techniques such as
`MPEG cause varying size frames, depending upon the difference from one frame to
`the next.
`
`
`MPEG Video Frames
`
`Figure 2. VBR video frames. Each frame contains a different amount of data, usually
`representing differences from the prior frame. The number of frames transmitted
`per second can also vary depending upon the complexity of the video stream.
` This technique reduces the storage space of the video. However, the number of bits
`needed to encode the video varies over time. This is known as VBR coding because
`the number of bits per second varies. (See Figure 2 above.)
`
`
`
`
`
`Instant Video Technologies, Inc.
`
` CONFIDENTIAL
`
`Page 4
`
`PAGE 4 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
` Typical state-‐of-‐the-‐art approaches do very little in the way of buffering at the
`receiver (client) computer. This means that any loss on the network will result in
`failure to maintain the frame rate. On small local area networks this is rarely a
`problem. However, as networks are layered, and made larger (e.g., the Internet) it
`becomes very difficult to provide meaningful service guarantees.
` Burstware ™ goes beyond typical state-‐of-‐the-‐art approaches to solving these
`problems, and takes a much more active approach. Our first step is to equip the
`video clients with several seconds worth of buffer. We then use a network channel
`with a bit capacity greater than the average video consumption rate. In addition, we
`use a server with enough capacity to monitor and anticipate the needs of the clients
`that it is serving.
`
`Network to Server
`
`Client System
`
`Big Bursts
`
`From Server
`
`To Server
`
`Status Info
`
`Burst Buffer
`Memory
`
`MPEG Video Frames
`
`
`
`Figure 3. Small status information packets are sent from the client (right) to the
`server (left) to indicate the state of the Burst Buffer Memory. The server sends
`bursts of data to the Burst Buffer Memory based on client needs. The client delivers
`VBR video frames from the Burst Buffer Memory as needed to support video
`playback.
` The Burstware ™ client sends the Burstware ™ server a series of status packets at
`regular intervals. (See Figure 3 above.) These packets inform the server as to the
`fullness of the client's buffer. From this information the server can derive the
`following:
` 1. network latency to the client
`2. bandwidth to the client
`3. how much space the client has for new data
`4. the client's average data consumption rate
`5. how long the client can run before it has no more video to display
`Instant Video Technologies, Inc.
` CONFIDENTIAL
`
`Page 5
`
`PAGE 5 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
` The server sends bursts to the client as appropriate, based upon scheduling
`constraints imposed by other work that the server must do, and prevailing
`conditions on the network. The server performs this task for many clients
`simultaneously.
`
`CLIENT 1
`Burst Buffer
`Memory
`
`VBR Video Stream
`
`Video
`Monitor
`
`CLIENT 2
`Burst Buffer
`Memory
`
`VBR Video Stream
`
`Video
`Monitor
`
`Video Bursts
`
`Buffer Status
`
`Video Bursts
`
`Buffer Status
`
`Video Bursts
`
`Video Server
`
`CLIENT 'N'
`Burst Buffer
`Memory
`
`Buffer Status
`
`Figure 4. A video server tracks information on and serves video to multiple clients.
`Each client is supporting a separate video monitor with its own video stream.
`
`VBR Video Stream
`
`Video
`Monitor
`
`
`
`Instant Video Technologies, Inc.
`
` CONFIDENTIAL
`
`Page 6
`
`PAGE 6 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
` Burstware ™ is implemented as a layer of software between TCP/IP and the client
`application on the client side, and as a similar layer between TCP/IP and the server
`application on the server side.
`
`Client
`Application
`Layer
`
`Video output
`File selection
`Play/pause
`
`Client
`Network
`Layer
`
`Send status
`Receive video
`
`Server
`Network
`Layer
`
`Receive status
`Send video
`Manage many
`Open files
`
`End User
`Video output
`Keyboard input
`
`ATM or Ethernet
`
`MPEG
`input buffer
`
`video data
`buffer
`
`Server
`Application
`
`File access
`Schedule service
`
`client list
`file handles
`file place
`channel bw
`channel lat.
`buffer stat...
`
`Server file system
`
`API
`
`TCP/IP
`sockets
`
`TCP/IP
`sockets
`
`API
`
`
`
`Figure 5. This figure shows a software scheme of one implementation of Burstware
`™. In this example, the Client and Server Network Layers interface with each other
`through TCP/IP sockets which can be utilizing any physical network topology (ATM
`and Ethernet are examples). The Client and Server Network Layers service their
`respective applications through API interfaces.
` This approach allows Burstware ™ not to involve itself with network hardware
`issues. Instead, it allows for the use of standard software, such as TCP/IP, to deliver
`packets. Another view of this architecture showing multiple clients is shown in
`Figure 6 on the following page.
`
`
`Instant Video Technologies, Inc.
`
` CONFIDENTIAL
`
`Page 7
`
`PAGE 7 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
`
`
`Athena Burstware
`System Diagram
`
`CLIENT
`
`User interface
`
`multimedia output
`
`SERVER
`
`Multimedia stream
`
`Multimedia filesystem
`
`multimedia files
`
`Burstware
`
`burst multimedia out
`status information in
`
`TCP/IP
`
`TCP/IP packets
`
`Network
`
`signals over
`ATM, Ethernet, ISDN...
`
`multimedia data
`
`Burstware
`
`burst multimedia in
`status information out
`
`TCP/IP
`
`TCP/IP packets
`
`Network
`
`CLIENT
`
`User interface
`
`multimedia output
`
`Multimedia stream
`
`multimedia data
`
`Burstware
`
`burst multimedia in
`status information out
`
`TCP/IP
`
`TCP/IP packets
`
`CLIENT
`
`CLIENT
`
`CLIENT
`
`CLIENT
`
`Network
`
`Figure 6. This figure shows the various software layers and the communication
`between them in a multiclient Burstware ™ video system.
`
`
`
`Instant Video Technologies, Inc.
`
` CONFIDENTIAL
`
`Page 8
`
`PAGE 8 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
`3.0 Environments in which Burstware ™ is Important
`
`what new situations demand the use of new protocols for data transmission such as
`
`In the metaphor mentioned in Section 1.2, the IVT Burstware ™ layer can be thought
`of as a bucket with an adjustable spigot. Large bursts of video are dropped into the
`top of the bucket from time to time, while video pours out of the bottom at an
`irregular pace (VBR). The server is watching lots of buckets that are all, potentially,
`at different distances and of different sizes and flow rates. The server
`communicates from bucket to bucket, keeping each bucket as full as possible, and
`without letting any one bucket go empty.
` The greatest benefit of Burstware ™, relative to other network approaches, is that of
`bandwidth optimization. Other approaches open up a channel of a certain
`bandwidth to each client; this assures each client that it will get what it needs. This
`approach, however, is terribly wasteful. VBR-‐coded video varies in bandwidth by a
`factor of at least five. In order to guarantee successful transmission, channel
`approaches must open a channel for the worst-‐case demand. By using a burst
`approach with central planning, the entire bandwidth of the network becomes
`available as needed. Clients whose video streams are not especially demanding of
`bandwidth are allocated only what they need, thereby freeing up bandwidth for
`more demanding streams.
`
`Networked computer systems have existed for many years before Burstware ™. So
`Burstware ™? In the following sections, we will review different systems
`environments that demand new techniques. In all cases we will be referring to
`applications involving the delivery and displaying of multimedia content material.
`
`In situations where latency is large with respect to bandwidth, protocols such as
`Burstware ™ have much to offer. As discussed in Section 1.1, the number of bits in
`the pipeline is approximately the latency times the bandwidth. This can present a
`problem on shared channels of high bandwidth and conventional latency. On
`dedicated channels, however, there must be adequate buffering in the protocol
`stack, or the full bandwidth will never be realized, with or without Burstware ™.
`
`It is common that networks with more than one switching element will have varying
`latency between points. This occurs when congestion at an output port causes a
`stream to be momentarily delayed until traffic clears. Also, in large scale networks,
`the path between points can arbitrarily change. By and large, as switched networks
`become more congested, latency fluctuates as the packets are routed through the
`switches.
` By its nature, Burstware ™ creates a large reservoir of data on the client side that
`allows the client to draw continuously even though the network latency varies.
`Burstware ™ makes an estimate of the network latency, and its
`Instant Video Technologies, Inc.
` CONFIDENTIAL
`Page 9
`
`3.1 Latency versus Bandwidth
`
`3.2 Variable Latency
`
`PAGE 9 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
`3.3 Variable Server Response
`
`expected variability. As long as the latency variations are not substantially greater
`than those predicted, the application program will not experience any delays.
`
`Just as with the variable latency case discussed in section 3.2, a conventional file
`server, such as an NFS (Network File System) server will experience variability in
`response time due to other overhead tasks. In general, computer file servers are not
`designed to provide high quality service all the time. They only have to deliver high
`quality service at an average acceptable for most data purposes. For this reason,
`there are companies producing "video servers." These servers are designed to
`deliver high quality service all the time. These servers, however, have to operate in
`a separate network environment. Burstware ™ provides a solution to multimedia
`file service over existing network environments.
`
`Probably the most complex aspect of current digital video technology is introduced
`by the success of video compression. Uncompressed video is CBR because it runs at
`30 frames per second with a constant number of bits to represent each frame.
`Compression schemes such as motion JPEG or MPEG compress the video stream
`depending upon content. Complex video streams take more data to represent than
`simple ones. Without getting into the details of video compression, it should still be
`clear that the data bandwidth demands of compressed video will vary over time.
` Any scheme that allocates a fixed bandwidth to a channel between server and client
`will not be able to take advantage of variable bit rate (VBR) coding. The channel will
`have to be allocated for the highest possible bit rate. Bit rates in VBR-‐encoded video
`can range over a factor of ten. By using faster-‐than-‐real-‐time bursts into large client
`buffers, Burstware ™ takes advantage of VBR-‐encoded video efficiencies. Long
`sections of uncomplicated video will consume less server and network resources
`than complex scenes.
`
`Burstware ™ co-‐exists quite effectively with other protocols on a network. With
`Burstware ™, one does not have to dedicate the entire network to the multimedia
`application. This is because Burstware ™ uses the resources that it finds on the
`network, and is constantly re-‐evaluating which resources are available. One
`concern when using a mixed-‐use network for the delivery of multimedia objects is
`that there can be sudden demands placed on the network which are difficult to
`anticipate. Reasonable care should be taken with respect to the network behavior
`when the network is being used for real-‐time purposes and conventional bulk
`transfers simultaneously. For most reasonably sized networks this is not a problem.
`However, if very large files are regularly moved around and large numbers of
`Burstware ™ users are to be supported, then some care must be exercised in
`network configuration.
`
`Instant Video Technologies, Inc.
` CONFIDENTIAL
`Page 10
`
`3.4 Variable Bit Rate Coding
`
`3.5 Mixed-‐Use Networks
`
`PAGE 10 of 13
`
`PETITIONERS' EXHIBIT 1009
`
`
`
`3.6 Internet
`
`4.0 Problems with ATM
`
`The Internet provides a very interesting and complicated environment for
`Burstware ™ protocols. It is not at all uncommon for there to be more than ten hops
`between any two points on the Internet. Each one of these hops has different
`bandwidth, latency and traffic congestion constraints. For this reason, the
`performance between any two sites on the Internet is highly variable moment by
`moment. It might well be argued that the Internet is not a suitable environment for
`serving multimedia objects. Suitable or not, however, the World Wide Web has
`created demand for such services on the Internet.
` Products such as Burstware ™ are essential to serving multimedia objects on the
`Internet. The underlying bandwidth must be faster than real-‐time and the client's
`buffering scheme must be able to deal with frequent, momentary reductions in
`bandwidth and increases in latency. While a greedy scheduling system on the client
`would work, such a system does not scale well. The aspect of Burstware ™ that
`centralizes scheduling in the