ISE1'42L
`
`5O5488 "CT 1 31928-
`
`PATENT DATE
`
`FILING DATE CLASS
`
`PATENT
`NUMBER
`SUBCLASS
`
`(10
`
`11 lite*(cid:9)
`
`, [
`
`GROUP ART UNIT
`;1445'
`2
`
`EXAMINER
`
`Ar(cid:9)
`
`Foreign(cid:9)
`claimed
`priority
`I:1 yes
`35 WC 119 conditions met(cid:9) 0 y(cid:9)
`no
`Verified and Acknowledged(cid:9)
`Examiner's Initials
`
`AS
`FILED
`°PIO'
`
`STATE OR
`COUNTRY
`',I(cid:9)
`'
`
`SHEETS
`DRWGS.
`
`TOTAL
`CLAIMS
`
`INDEP.
`CLAIMS
`
`FILING FEE
`RECEIVED
`
`ATTORNEY'S
`DOCKET NO.
`
`SS TErn F s. asT_Tic tr.(cid:9)
`ETRIEVPIL OF FLwTt JnTdjfl
`TR-nc,- On-re P1ictiei::
`LES V eR- COrn PU-T ER. A..)E-1-4 )eR_LS(cid:9)
`tPrT TP-Pl■SSPCSST15.3(cid:9)
`PtT E71-)k--:Therettn) Eb tar*r. gPe6grYRT.1P-4313L (Rev.12-94;
`
`PARTS OF APPLICATION
`FILED SEPARATELY(cid:9)
`NOTICE OF ALLOWANCE MAILED
`
`n
`ISSUE FEE
`ilt7oPaici) e (cid:9)
`AmPirlt Due
`$660,00 (cid:9)
`
`Label
`Area
`
`Form PTO-436A
`(Rev. 8(92)
`
`Applications Examiner
`CLAIMS ALLOWED
`Print Claim
`Total Claims(cid:9)
`4.z
`DRAWING
`Sheg Drwg. Fig5.1t
`2
`
`ISSUE(cid:9)
`BATCtI;
`NUMBER
`
`Print Fig.
`
`ant Examiner
`
`(cid:9)/1%1
`
`iiettj
`M STAM M. MEKY
`PRIMARY MINER
`
`Primary Examiner
`PREPARED FOR ISSUE
`
`WARNING: The information disclosed herein may be restricted. Unjistforized disclosure may be prohibited
`by-the United States Code Title 35, Sections . 122, and 368. Possession outside the U.S.
`Patent & Trademark Office is restricted to authprted employees and contractors only.
`
`ISSIir rrr
`
`ChenFH001
`
`PAGE 1 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`(cid:9)
`

`
`Date
`Received
`Mailed
`
`(cid:9)9 (0
`9
`/ - 7 - ? 7 a "14i - J
`1 -7 - 17
`
`---1-)
`cc--it--7r)
`
`4-2,/-9si
`
`4-01i-gg
`
`4-,./--92
`
`5 /7(9S
`
`PATENT APPL 1CM ION
`
`.UI1IHI 1101 till IilIt%I 1101111
`08505488
`
`CONTENTS
`
`1. Application
`
`-'3 Ex I-(cid:9)
`
`et 7744 e ( i 014 OS :)(cid:9)
`
`Re.vi,,artc.s
`
`( Recon.) el- -becl.caterfrion
`
`Date
`Entered
`or
`Counted
`
`C
`
`-
`
`O
`
`22. (cid:9)(cid:9)
`
`1,6://;e-c-larboi-icsn
`_.„141.1 l'4.,) -,r- 'R€0. (xi- :-(cid:9)
`., . _ . 1
`
`corn-v4 rnwi72s
`P10 GRANT OCT 1 a 1998
`
`21.
`
`• 23.
`
`24.
`• (cid:9) 25.
`
`26.
`
`27.
`
`28. (cid:9)
`
`29.
`
`30.
`
`31. (cid:9)
`
`32.
`
`ChenFH002
`
`PAGE 2 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`
`SEARCHED
`
`Exmr.
`
`dikt_
`
`Jr,
`200.33
`2c0,11 7
`200,4 9
`200.6i
`2.0 0776
`
`34 6
`
`Iq
`
`H'iH
`
`it4
`
`SEARCH NOTES
`
`Date
`
`Exmr.
`
`4P6 (so t(04
`cc/Ina /ont)
`
`PI/cm
`P 4
`(ertitfp,
`Pc#240).
`
`•
`
`APc
`
`INTERFERENCE SEARCHED
`Sub.
`Exmr.
`Date
`Class
`a0 0 ' 33 SOPS P1,1114
`
`395
`
`ChenFH003
`
`PAGE 3 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`Staple Issue Slip Here
`
`ID NO. (cid:9)
``19
`
`CYTh
`
`DATE
`S/4/95-
`
`POSITION
`CLASSIFIER
`
`EXAMINER
`
`TYPIST
`
`VERIFIER
`CORPS CORR.
`SPEC. HAND
`FILE MAINT.
`DRAFTING
`
`INDEX OF CLAIMS
`
`Claim
`To
`
`Date
`
`10
`
`Claim
`
`1111111•
`nreciummann
`3
`4
`5
`
`to a
`
`•
`
`I61II
`
`05
`
`1
`52
`53
`54
`55
`56
`57
`58
`59
`60
`61
`62
`63
`64
`65
`66
`67
`68
`69
`70
`71
`72
`73
`74
`75
`76
`77
`78
`79
`80
`81
`82
`83
`84
`85
`86
`87
`88
`89
`90
`91
`92
`93
`94
`95
`96
`97
`98
`99
`100
`
`Rejected
`Allowed
`
`Restricted
`Non-elected
`nterterence
` Appeal
`Objected
`
`. (Through nomberal) Canceled
`11111.1111111 (cid:9)
`
`SYMBOLS
`
`O (cid:9)
`
`SIM
`
`El 39 111115111111111MIIMI
`12 11101111111111111•11•
`1111311111MIIIIIIMISIN
`42 j
`43
`44
`45
`46
`47
`48
`49
`50
`
`ChenFH004
`
`PAGE 4 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`
`PATENT APPLICATION SERIAL NO. 505488
`
`U.S. DEPARTMENT OF COMMERCE
`PATENT AND TRADEMARK OFFICE
`FEE RECORD SHEET
`
`300 SC 08/23/95 08505488 Ok_.
`1 201(cid:9)
`683.00 CN
`
`07-1135 180 201(cid:9)
`
`38.00CR 0247,(70
`
`SE18015 09/22/95 08505488
`
`PTO-1556
`(5/87)
`
`ChenFH005
`
`PAGE 5 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`CCetlISSICKER cc
`haVIDCIEN, O.C. 2
`Sir:
`
`505488
`
`For:
`
`Trartsnittal herewith
`is the patent application of
`Imitator: HUEY SHIANG CHEN, MON-SONG CHEN, SHIOW-LAANG HUANG and DEYANG SONG
`"A METHOD FOR JUST-IN-TIME. RETRIEVAL OF MULTIMEDIA FILES
`OVER COMPUTER NETWORKS"
`EnClOsCd are:
`three (3) (cid:9) sheets of drawing.. (informal)
`171] Wassitprnentof the invention to INFOVALUE COMPUTING, INC.
`along with' $40 check for recordal fee
`
`A certified copy of a application.
`Ii Art associate Awes of attorney.
`A verified statement to establish snail entity status under 37 CFR 1.9 and
`37 CM 1.27.
`
`ii
`
`The filing fee has bee) calculated as stain below:
`
`(Col. 1)
`C. FILED
`
`(Col. 2)
`C. ECU
`
`FOR:
`sksx fl
`
`22
`
`:*.'CAL CAD15
`
`4 2-213`
`5 -3=
`DCEP. CALMS
`D M1-12IPLE OUDELV2 CALM PRSEDCED
`If the difference in Col. 1 Is less
`than zero, enter • 0" 14-1 Col.
`
`SNAIL atrITY
`
`OMER THAN A
`SKALL, Emil?
`
`X 1 1
`x38
`X $120
`TOTAL.
`
`$365
`5242
`$(cid:9) 76
`S
`
`5.683
`
`in the rant of $
`
`P:ease d:arge Try Deposit escort No. •
`I
`A duplicate cosy of this sheet is enclosed.
`bD A check in*otZarcunt of S 72 3 2 .0 * to cover the . filing fee is enclosed.
`The Commissioner itgheegyalitifitijiMntafeaa fai3trgniin
`IX J fees associated with this corthrtication or credit any overpayment to
`Deposit Act-I:twit No. 07-1135 (cid:9) . A duplicate copy of this sheet is enclosed.
`Any, additional filing fees required under 37 CFR 1.16.
`'My patent application processing fees under 37 CFR 1.17.
`The' Connissioner is hereby authorized to charge payment of the following fees -
`duritq the pendency of this application or credit any 'overpayment to
`.. A duplicate copy of this sheet is enclosed.
`Deposit . Accazt t4o.
`ED My patent application processing fees under 37 CFR 1.17.
`of the7
`The issue fee set forth th 37 CFR 1.18 at•or before
`Notice of Allowance, pirsuant to.37 CM 1.31190.
`–1 My filing fees under 37 CFR 1.16 for presentation of ectra
`
`,------
`ELIOT S. GERA-Reg. #18,115 (
`wwwrigERBER, BURKE(cid:9) =
`AttornèyfptCApplicant BA&
`
`;
`
`ChenFH006
`
`PAGE 6 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`
`
`(cid:9)
`

`
`505488
`
`CERTIFICATE OF MAILING BY EXPRESS MAIL
`
`EXPRESS MAIL Mailing Label No. EM243673146US
`Date of Deposit:(cid:9)
`July 21' 1995
`
`I hereby certify that the accompanying papers comprising Application For
`Letters Patent entitled:
`"A METHOD FOR JUST-IN-TIME RETRIEVAL OF MULTIMEDIA FILES
`OVER COMPUTER NETWORKS"
`are being deposited with the United States Postal Service "Express Mail Post
`Office to Addressee" service under 37 CFR 1.10 on the date indicated above
`and are addressed to the Commissioner of Patents and Trademarks;
`Washington, D. C . 20231.
`
`Dated: 9„,(cid:9)
`
`/99,-
`
`Ire t Bieniewicz
`
`e
`
`ChenFH007
`
`PAGE 7 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`ao IL
`505488
`
`A METHOD FOR JUST-IN-TIME RETRIEVAL OF
`MULTIMEDIA FILES (cid:9)
`
`Field Of the Invention
`The present invention relates to methods for retrieval of
`multimedia files, such as video, over computer networks.
`
`Background Of The Invention
`Multimedia computing has recently emerged as an important
`information technology. Multimedia computing obtains information
`from a variety of media retained in information storage in the
`form of digital data, for example, motion video clips (segments)
`and audio clips. This allows businesses and non-profit
`organizations to create highly effective computer presentations
`and, using computers, provide superior training and education.
`For example, instead of storing paper technical manuals,
`which must be searched manually, workers on a manufacturing floor
`could use a computer terminal, called a "client machine", to
`interactively access a large collection of multimedia training
`materials stored in a centralized "server". A "client machine"
`is a computer or terminal having a screen that an individual uses
`to access and display video files. It has at least three
`component processes: MM (Multimedia) application, i.e.,-the
`display of a video clip; Client Agent, i.e., software (computer
`program to access the server) and Network Interface, i.e., an
`interface card. A server (server machine) is a computer, for
`example, a "mini-computer" or workstation computer or super-
`
`ChenFH008
`
`PAGE 8 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`computer, connected to a number of computer terminals (client
`machines) in a Local Area Network (LAN) or other computer
`networks, such as metropolitan area networks, i.e., FDDI and high
`speed wide areas networks, such as ATM. The server stores and
`delivers digital video clips to multiple client machines in a
`network. The server includes, at least, three component process:
`Network Interface, i.e., an interface card; Server Control, i.e.,
`software to control access and delivery of stored video clips;
`and Storage Subsystem, i.e., digital memory storage of video. In
`interactive access, the user operates his client machine to
`request multimedia files from the server for display on the
`screen of the client machine and the server responds to the
`user's requests. For example, a worker, at the press of a key,
`could have the server retrieve more detail, on the subject of his
`choice, using text and video. In this arrangement, "a centralized
`server (as opposed to an individual computer) stores the
`necessary multimedia information. This arrangement exists
`because video clips (usually video .1 to 10 minutes long) require
`a large amount of computer storage. For example, a one-minute
`video clip, compressed, may be over 12,000,000 bytes (12 Mb - a
`byte here being 8 digital bits) in size. Thus, having a large
`selection of video clips on the individual computers of a network
`would not be economical. Multimedia technology may also provide
`new forms cf entertainment, video-on-demand being one well-known
`example. In a video-on-demand system, a centralized networked
`
`-2-
`
`ChenFH009
`
`PAGE 9 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`storage server stores a large collection of videos, such as
`entire full-length feature films in video form (90-180 minutes).
`A plurality of users may simultaneously retrieve their preferred
`video features at their selected viewing times.
`Multimedia computing has significantly greater processing
`power and storage requirements than computing that only involves
`text and numeric data. Typical motion video, for example, has
`data rates (rates of information flow) well exceeding 100 Mbps
`(million bits (megabits) per second), without compression, and
`between 1 to 8 Mbps with video compression. Video compression
`uses algorithms to pack video data in a form taking up less
`memory and requiring less transmission speed. It strives to make
`the data look realistic when it is .played back on a computer
`monitor screen or TV set. "Video playback boards" are hardware
`used to decompress data and utilize various compression
`standards. These compression standards, which vary in their Mbps
`utilization, govern the viewing quality of videos. Compression
`standards include Indeo (1.2 Mbps), MPEG-1 (Motion Picture
`Experts Group-1) (1.5 Mbps), motion JPEG (Joint Photographic
`Experts Group) (5-6 Mbps) and MPEG-2 (Motion Picture Experts
`Group-2). (6-8 Mbps).
`In most situations, only compressed video can be stored and
`delivered across networks due to the limits of transmission speed
`and computer memory. Even with compression, however, video still
`has a high memory storage requirement, and thus is ideal for
`client server configurations in a data network. In those
`
`ChenFH010
`
`PAGE 10 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`configurations, a server holds compressed video clips although,
`less preferably, it may hold uncompressed video clips and
`compress the video as it transmits it. Many client machines,
`which individuals control, can access the server and obtain the
`video clips in compressed form. The client machine then
`decompresses the video clips and displays them on its screen for
`the user to view. Hence, the high cost of installing expensive
`large capacity storage in all client machines is avoided.
`However, transmitting multimedia data across computer
`networks for immediate playback at the receiving end is
`complicated by both the nature of computer networks and the
`elaborate processing necessary for effective playback. Today's
`prevalent computer networks use statistical multiplexing for
`transmission. This process "packetizes", i.e., divides data into
`segments so that it can be transmitted within and between
`computers. The data, in packet format, can then be transmitted
`through a series of store-and-forward operations, because the
`packet contains a "header", which is a set of bytes identifying
`the packet and usually identifying its destination, i.e., the
`identity of the computer to which it should be transmitted. Each
`time a networking computer stores-and-forwards (puts into its
`memory and then transmits to another computer) an additional
`element of variable delay is added to the end-to-end transmission
`latency (the time it takes to move a packet from one end of the
`network to another). The variability of this store-and-forward
`
`-4-
`
`ChenFH011
`
`PAGE 11 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`delay depends on the number of other packets that are queued
`(waiting in line) and competing for the same transmission
`resources. Due to this variability, different packets of the
`same video stream experience differential transmission latency
`(different amounts of delay). This differential latency causes a
`phenomenon known as "delay jitter", which the viewer sees as
`jerky motion and inferior or unacceptable picture quality.
`The prevailing approaches to transmission of video over
`computer networks focus on transmitting the same average rate of
`data to the client machines from the server. The average rate,
`in bytes per second, is computed by dividing a file's total size
`by its playback length (in seconds). This average data rate,
`however, only indirectly measures motion smoothness, because
`video frame sizes may vary by as much as a factor of 10. In
`MPEG-1 (Motion Picture Experts Group), a video compression
`standard, the size of an I (Intra) frame, on average, is about 2
`to 3 times that of a P (Predictive) frame, and the size of a P
`frame is about 4 to 5 times that of a B (Bi-directional
`interpolated) frame.
`Due to their isochronous property (regular timing of
`a stream), transmission of multimedia data files requires
`consideration of factors that traditional data transport
`protocols, such as TCP/IP, do not consider. Transmission.
`of video requires that frames are played back at fixed
`intervals (fixed time frames) to ensure motion smoothness and
`thus viewing quality. Traditional systems such as TCP/IP
`
`-5-
`
`ChenFH012
`
`PAGE 12 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`simply manage the data in blocks, addressing only throughput but
`not timing considerations. New proposals tailored for
`transmitting multimedia files focus on utilizing additional
`and/or stronger network functionality, such as priority
`transmissions and guaranteed bandwidth. Guaranteed bandwidth
`means that a transmission will always have up to a specific
`amount of information-carrying capacity reserved for its use.
`While these approaches provide some improvement, they still treat
`a multimedia file as a byte stream, and consequently aim to
`maintain an average transmission rate. Such an average
`transmission rate does not take into account the variability in
`individual frame size, i.e., how many bytes are in each frame.
`Given the large variations in frame sizes, frame rates and
`end-to-end transmission latencies, it is not efficient to treat a
`multimedia file simply as a byte stream. However, that is the
`conventional and prevailing approach which is presently being
`commercially implemented. Even though both video and text are
`transmitted in the form of digital data in packet format, video
`transmission is more problematic. To regulate the flow of video
`transmissions, the client machines and server machines should
`frequently exchange control messages. These messages should
`result in adjustments of the rate of data flow. A need exists
`for an efficient method for a client machine to retrieve
`multimedia data from a server with minimum latency (delay) and
`minimum overhead (use of memory and processing); with flexibility
`
`-6-
`
`ChenFH013
`
`PAGE 13 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`in satisfying a variety of application requirements; with the
`final picture on the client machine monitor being of high
`quality, i.e., not being jerky; and without the use of additional
`memory or other additional hardware.
`
`Summary Of The Invention
`The present invention provides a method for a client machine
`to retrieve multimedia data from a server machine with minimum
`latency and overhead. The method is preferably implemented by
`installing software programs on the client machines and server,
`without any hardware changes; assuming that the client machine
`and server have sufficient memory and the client machine has
`adequate video decompression capabilities.
`The method of the present invention also provides multimedia
`data as readily available to application programs as if that data
`were in the form of files in the data memory storage of the
`client machine. This invention minimizes the amount of buffer
`(memory space for temporarily storing multimedia data) in the
`client machine.
`Accordingly, this invention is a method for retrieving
`multimedia files from a server computer to a client computer over
`computer networks.
`
`-7-
`
`ChenFH014
`
`PAGE 14 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`In the preferred embodiment the server transmits the
`multimedia file in the form of digital data packets. The
`transmission rate is based on the frame rate of the file. There
`is no attempt to provide an even data stream based on averaging.
`First, there is provided a means for extracting essential
`timing information which has been specified in the multimedia
`file. Multimedia data streams have well-defined building blocks
`(objects) which should be the basis for transmission timing in
`order to insure playback quality. In MPEG-1 motion video (Motion
`Picture Experts Group-1), a video compression standard, the video
`picture or frame serves as the building block. Although a
`constant number of video frames are played in 1 second, generally
`30, the amount of data in one second of video varies with the
`complexity of its content. For example, a video clip showing a
`bar graph has low complexity compared to a video clip of a
`football game. A-separate-aceempanying index-file stores timing
`information whichrepresents-the-COMPlexity of content of each
`frame.- A software-implementedproteSt ih-the- gerver-uses this
`index file to schedule transmission.H The transmission of the
`compressed-video-data -1SbaSed-ora direct measure-of-itS-timihg.
`That-direct measure-is based-up-on, and derived -from, the video
`frame__ K40_and_the_mideo_frame size.
`
`-8-
`
`ChenFH015
`
`PAGE 15 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`Secondly, the present invention uses the timing information
`in the index file to (i) ensure transmission of a video frame in
`a frame time under normal circumstances, e.g., 30 frames per
`second, and (ii) provide an explicit hand-shake protocol (a
`series of back and forth prearranged digital messages
`coordinating a set of operations) between client agent and server
`processes. This protocol regulates the transmission rate in
`exceptional situations.
`Thirdly, the present invention provides a process in the
`client agent to receive data from the server with minimum latency
`(delay); with minimum overhead (use of memory and processing
`resources); and with the flexibility to satisfy a variety of
`application requirements. The process in the client agent has
`the intelligence to selectively execute the explicit processing
`required for additional features when such additional features
`are needed.
`Finally, this invention provides a process in the client
`machine to monitor and log the behavioral characteristics of the
`applications. The log enables the process in the client machine
`to fine-tune its processing during run time (the time it is
`operating) and optimize its interactions with the process in the
`server computer.
`This invention times the transmission of multimedia files
`according to a fixed rate, generally the frame rate during normal
`transmission. For example, if the client machine can display 30
`
`-9-
`
`ChenFH016
`
`PAGE 16 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`frames per second, the server will , transmit a frame of compressed
`video starting at each 1/30th second, regardless of the
`complexity of the video frame. The client machine needs to store
`in its memory only one, or a few, frames as new frames are
`transmitted to it at a regular rate (frame rate). However, if
`the data in the buffer of the client agent is below a selected
`standard ("water mark"), the transmission rate is increased; if
`above a selected standard it is decreased.
`
`Brief Description Of The Drawings
`Figure 1 is a schematic illustration (block diagram) of the
`client machine retrieving multimedia data from the server machine
`over a computer network;
`Figure 2 is a schematic illustration of the structure of the
`client agent;
`Figure 3 is a schematic illustration of the buffer
`management in the client agent;
`Figure 4 is a schematic illustration of the desirable frame
`level pacing based on the essential timing information specified
`in multimedia files;
`Figure 5 is a schematic illustration of the design of the
`server; and
`Figure 6 is a schematic illustration of the transmission
`scheduling algorithm.
`
`-10-
`
`ChenFH017
`
`PAGE 17 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`Description Of The Preferred Embodiments
`Figure 1 represents schematically the overall system. The
`client machine (20) is the computer upon which a user types his
`commands, for example, a PC (Personal Computer) which may have a
`relatively low end integrated circuit microprocessor such as an
`Intel (TM) 386 processor, although any type of PC may be used.
`The user wishes to retrieve multimedia files from the server (21)
`via data connections and over a computer network. The client
`machine (20) has three interacting processes: the client agent
`(30) which interfaces with the network interface (3) and the
`multimedia application (4) in the client machine. For example,
`the server (21) may be a workstation such as a Sun (TM)
`workstation or IBM PC Server 300 and having a high throughput and
`storage capability, for example, using a disk array. A typical
`multimedia application is the playback of a full-motion video
`clip. The network, for example, may be an Ethernet (bus network
`topology) which may be implemented with coaxial wiring and 1000-
`3000 feet between nodes, or a Token Ring system (high speed token
`that checks in at each node, available from IBM).
`The client agent (30) has the primary responsibility of
`retrieving from the server control (1) the right set of
`multimedia data at the right time to satisfy the needs of the
`multimedia application (4). The client agent (30) maintains a
`packet buffer (31) (a structure for temporary data storage) as a
`cache storage (temporary data storage center). For example, the
`
`ChenFH018
`
`PAGE 18 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`packet buffer (31) may be a section of the RAM (Random Access
`Memory) of the PC. Correspondingly, the primary responsibility of
`the server (21) is to read from disks and make just-in-time
`delivery of the appropriate set of multimedia data. The server
`(21) has three component processes: the server control (1) which
`interfaces with the storage subsystem (12) and with the network
`interface (2). Similar to the packet buffer (31) in the client
`agent (30), a stream buffer (11) in the server control (1) holds
`the data that has been read from the storage subsystem (12). The
`stream buffer (11) serves as a temporary data storage center for
`the server control (1).
`The interactions between the server control (1) and the
`client agent (30) go through the network connecting the two
`machines. The network interface (2) in the server (21) and the
`network interface (3) in the client machine (20) support network
`connectivity. Specifically, the present embodiment uses two
`logical connections. The control channel (5) serves to exchange
`control messages. The data channel (6) serves to transmit
`multimedia data from the server (1) to the client agent (30).
`One possible implementation would use a reliable TCP protocol
`line for the control channel, and a fast and mostly reliable UDP
`protocol for the data channel.
`Figure 2 schematically represents the preferred detailed
`structure of the client agent (30). As depicted in Figure 1, the
`client agent (30) interfaces with the network interface (3) and
`
`-12-
`
`ChenFH019
`
`PAGE 19 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`the multimedia application (4). Two execution paths exist, one
`for data and one for control messages. The data execution path
`starts from the data receiver (32), which receives incoming data
`packets from the network. Then the data receiver (32) signals
`the buffer manager (38) to place the data packets properly into
`the structure of the packet buffer (33). The application
`interface (35) accesses the multimedia application 4 and
`translates its commands to the client controller 36. The output
`processor (34) delivers data to the multimedia application 4.
`The packet buffer (33) stores data packets until the multimedia
`application requests that they be delivered to the multimedia
`application (4). If the packet buffer (33) does not have the
`requested data available, the client controller (36) signals the
`command processor (37) to send .a command packet (a packet of
`information making a specific request) to the server control (1
`in Figure 1) for immediate retrieval of the requested data. The
`command processor (37) sends out the command packet via the
`control channel (5 in Figure 1) and through the network interface
`(3).
`
`The buffer manager (38) manages the structure of the data in
`the packet buffer (33). Figure 3 describes in detail the
`structure of the packet buffer (33). As to the amount of data,
`ideally the packet buffer (33) should have enough data: (i) to
`minimize the possibility of not having the requested data, and
`(ii) still have enough free buffer space (memory space) to
`
`-13-
`
`ChenFH020
`
`PAGE 20 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`receive new data packets. "Water Marks" regulate the server's
`transmission rate thereby balancing between these two conflicting
`factors. Three transmission modes are defined: NORMAL, RUSH,
`and PAUSE. Based on the amount of data in the packet buffer (33)
`the client agent (30) decides which is the appropriate mode.
`When a change occurs, the client agent . (30) informs the server
`control (1). The client agent (30) changes the transmission mode
`based on a series of rules, explained below.
`To understand these rules, one must first understand the
`"Water Mark" model. This model draws a parallel between the
`client agent buffer and a water bucket with a spout at the bottom
`that brings water to a person. Water entry (Application Data
`Unit, ADU entry) occurs intermittently. For example, other
`network traffic could slow ADU entry, or a collision sequence in
`the Ethernet, could stop it entirely. Assuming the bucket
`(client agent packet buffer) never empties, water exits from the
`spout at a constant rate. Continuing the bucket analogy, the
`bucket constitutes a set 'of frames in the packet buffer (33)
`which is the cache (temporary memory, generally RAM) used by the
`client agent. The bucket has high and lower "water marks". In
`the just-in-time retrieval method, when the amount of data falls
`between the water marks, transmission occurs in NORMAL mode. In
`the present invention the transmission should be in NORMAL mode
`most of the time. For example, the packet buffer will normally
`
`-14-
`
`ChenFH021
`
`PAGE 21 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`store 1-5 frames of video. In this mode the server (1) paces its
`transmission so that the data for a single video frame is
`transmitted in the time of .a single video frame (normally 1/30
`second), as Figure 6 will discuss in detail. Transmission occurs
`very efficiently in this NORMAL mode because no need exists for
`the client agent (30) to send periodic feedback to the server
`control (1).
`Transmission enters PAUSE mode when the amount of data
`exceeds the high water mark, i.e., there is too much data in the
`client agent packet buffer (33). Transmission occurs in RUSH
`mode when the amount of data falls below the lower water mark,
`i.e., there is not enough data in the client agent packet buffer
`(33). The client agent (30) sends a "NORMAL-TO-RUSH" command if
`the amount of data decreases below the low water mark. The
`client agent (30) sends a "NORMAL-TO-PAUSE" command if the amount
`of data increases above the high water mark. The client agent
`sends a "PAUSE-TO-NORMAL" command if the amount of data decreases
`from above to below the high water mark. The client agent (30)
`sends a "RUSH-TO-NORMAL" command if the amount of data increases
`from below the lower water mark to above the low water mark.
`Figure 3 schematically represents the structure of the
`packet buffer (33). Each data packet contains a packet header
`(53) and the multimedia data (60). The packet header (53) should
`contain at least the following five elements of information (54):
`
`-15-
`
`ChenFH022
`
`PAGE 22 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`Pkt. Seq. No: a unique packet sequence number
`Frame No.: the video frame number to which
`the data in the packet belongs
`InFrame Seq. No: the sequence number of the packets
`within the same frame, e.g., 1 is for the first
`packet of a video frame, 2 the second, and so on, and 0
`is the last packet of the frame
`Offset: file offset of the first data byte in the
`packet
`Size: the number of data bytes in this packet.
`The transmission scheduler sets these data during
`packetization, as Figure 6 discusses in detail. The data queue
`(52) organizes the packets by putting them in a specific order;
`the packets are sorted according to the "offsets" of the data.
`"Offset" is a number representing the relative position of a
`byte, generally in regard to the start of a file. The buffer
`manager maintains the packet structure until delivering the data
`to the applications. Two consecutive packets in the buffer need
`not have contiguous file offsets. For example, the user may
`select file portions which are not in order, i.e., one minute of
`video from the start of a video clip followed by one minute from
`the end of the clip. Therefore the file offset must be
`explicitly checked before delivering a data packet to the
`multimedia application (4). Otherwise an incorrect set of data
`may be delivered to the multimedia application (4).
`
`-16-
`
`ChenFH023
`
`PAGE 23 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`Fast and mostly reliable UDP-like channels transmit data
`packets. At some network nodes, packets may be lost, for
`example, due to line noise or buffer overflow. In one error-free
`embodiment the lost packets are traced and replaced and in
`another embodiment, not error-free, there is no attempt to
`replace lost packets.
`To detect lost packets, in an error-free embodiment, the
`client agent (30) uses a register to maintain a variable Last
`Pkt. Seq. No. (51), which is the packet sequence number of the
`last received packet. If the Pkt. Seq. No. of the newly arriving
`packet denoted as New Pkt Seq No differs from (Last Pkt. Seq. No.
`+ 1), then a packet loss has occurred. Specifically, the packets
`with Pkt. Seq. No.'s from (Last Pkt. Seq. No. .+ 1) to (New Pkt.
`Seq. No. - 1) have been lost.
`To deal with packet loss, the client agent (30) maintains a
`list of lost packets (56) in a linked list or other data
`structure. That list records the two most important pieces of
`information about the lost packet, namely, its Pkt. Seq. No. and
`Time Out Value (57). When the client agent (30) sends the
`"retransmission request" for lost packets to the server control
`(1) the Time Out Value is set. If the missing data packet
`arrives correctly before the Time Out Value expires, this removes
`that data packet from the list. If not, the client agent (30)
`
`-17-
`
`ChenFH024
`
`PAGE 24 OF 216
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`(i) either sends another "retransmission request" to, the server
`control (1) or (ii) gives up on obtaining the missing data packet
`and removes its number from the lost packet list.
`While providing data to the multimedia applications (40) the
`client agent (30) also monitors the characteristics of those
`applications. The frametime (55) and Ave. Data Rate (50)
`registers record the application's two most important
`characteristics, which are (i) average frame time and (ii)
`average data rate. In one embodiment, such monitoring is
`executed for each frame, i.e., executing the monitoring when the
`delivered packet has In Frame Seq. No. equal to zero. If these
`two average data rates differ significantly from the expectation
`of the server, which is transmitted to the client agent (30), the
`client agent (30) sends the corresponding commands, i.e., Frame
`Time Req. and Data Rate Req. to the server control (1).
`To support regulation of the transmission pace with the
`amount of data in the packet buffer as described in connection
`with Figure 2, the client agent (30) also maintains the Tx. Mode
`(58) and Data Queue Size (59) registers.
`The design of the client age