(12) United States Patent
`Carmel et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,389,473 B1
`May 14, 2002
`
`US006389473B1
`
`(54) NETWORK MEDIA STREAMING
`
`(75)
`
`Inventors: Sharon Carmel; Tzur Daboosh, both
`of Giv’atayim; Eli Reifman, Rishon le
`Zion; Naftali Shani; Ziv Eliraz, both
`of Tel Aviv; Dror Ginsberg, Karkur;
`Edan Ayal, Kfar Saba, all of (IL)
`
`(73) Assignee: Geo Interactive Media Group Ltd.,
`Givatayim (IL)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/275,703
`
`(22)
`
`Filed:
`
`Mar. 24, 1999
`
`(30)
`
`Foreign Application Priority Data
`
`Mar. 24, 1998
`
`(IL)
`
`.............................................. .. 123819
`
`Int. Cl.7 .............................................. .. G06F 13/00
`(51)
`
`. . . . . . . . . . . . . .
`(52) U.S. Cl.
`. . . . . . . . . . . . . . . . . . . . . . .. 709/231
`(58) Field of Search ..................... .. 707/500.1; 709/200,
`709/231, 236, 246, 247, 382/236, 239,
`375/240.12
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,267,334 A
`
`11/1993 Normille et al.
`
`.......... .. 382/236
`
`5,404,446 A
`5,841,432 A
`
`4/1995 Bowater et al.
`11/1998 Carmel et al.
`
`.......... .. 345/537
`......... .. 707/500.1
`
`Primary Exami/1er—Robert B. Harrell
`
`(74) Attorney, Agent, or Firm—Ladas & Parry
`
`(57)
`
`ABSTRACT
`
`A method for real-time broadcasting from a transmitting
`computer to one or more client computers over a network,
`including providing at
`the transmitting computer a data
`stream having a given data rate, and dividing the stream into
`a sequence of slices, each slice having a predetermined data
`size associated therewith. The slices are encoded in a
`
`corresponding sequence of files, each file having a respec-
`tive index, and the sequence is uploaded to a server at an
`upload rate generally equal to the data rate of the stream,
`such that the one or more client computers can download the
`sequence over the network from the server at a download
`rate generally equal to the data rate.
`
`41 Claims, 11 Drawing Sheets
`
`Microfiche Appendix Included
`(2 Microfiche, 138 Pages)
`
`
`
`STANDARD
`NETWORK
`SERVER
`
`
`
`28
`l
`
`HTTP
`
`
`
`//9 so
`.. '
`
`‘ T‘ ‘
`
`28
`
`
`
`PAGE 1 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 1 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 1 of 11
`
`US 6,389,473 B1
`
`mm
`
`I-
`
`‘<’:’
`095
`9E
`gm
`
`24
`
`REALTIME
`
`ENCODE
`
`/”
`
`Q
`
`<.'\J//
`C\l
`
`CD
`C'\I
`
`"'—'__05
`<C
`1
`(39‘K
`L|.0-
`
`PAGE 2 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 2 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 2 of 11
`
`US 6,389,473 B1
`
`N 9
`
`5
`
`LI.
`
`
`
`28
`
`‘
`
`S’:
`
`\\
`
`H
`
`
`
`.
`
`PAGE 3 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 3 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 3 of 11
`
`US 6,389,473 B1
`
`I-——T1—-I-—T_9_——l-—T3——!-——T4——l
`
`FIG. 3A
`
`SUCE
`1
`
`SUCE
`2
`
`SUCE
`3
`
`SUCE
`4
`
`42
`
`44
`
`46
`
`48
`
`
`
`TIME
`
`/
`
`40
`
`FIG. 3B
`
`
`
`
`
`SUCE N
`
`MESSAGE
`
`FIG. 3C
`
`
`
`PAGE 4 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 4 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`mm317_H49B9I8H3,a6SSu
`
`UUL
`
`2002
`
`11f04tCChS
`
`tHC
`
`1yaM
`
`49coo
`
`t1a2PmS.5oEUG
`
`000
`
`AD:
`
`SL
`
`PAGE 5 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 5 of 11
`
`US 6,389,473 B1
`
`4 F
`
`IG.
`
`K 0
`
`:31O>
`EE
`Zcn
`
`LO
`P’)
`
`$01
`(OLD
`
`(0003
`LDLOl‘~
`
`:—(\ll")fl'LOCO
`ZXXXXEC
`zzzzzz
`_.J._I_::1Z:|
`
`PAGE 6 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 6 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 6 of 11
`
`US 6,389,473 B1
`
`FIG. 5
`
`
`
` INPUT
`BROACAST
`DATA
`
`ENCODE
`
`
`
`PAGE 7 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 7 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 7 of 11
`
`US 6,389,473 B1
`
`FIG. 6A
`
`CONNECT
`
`TO SERVER
`
`HTTP FROM
`
`SERVER
`
`READ INDEX
`
`FILE
`
`SELECT SLICE
`
`
`
`CONNECT
`NEW LINK
`
`
`
`
`
`CONTINUE
`
`
`
`DECODE
`
`OUTPUT
`
`DATA
`
`
`
`
`
`PAGE 8 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 8 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 8 of 11
`
`US 6,389,473 B1
`
`FIG. 6B
`
`CONNECT
`
`TO SERVER
`
`
`
`
`HTTP FROM
`
`"SERVER
`
`
`
`
`
`
`
`
`CHOOSE
`
`LOWER
`AUDIO/VIDEO
`LEVEL
`
`CONTINUE
`
`
`
`READ HEADER
`
`CHOOSE
`
`LEVEL
`
`DECODE
`
`OUTPUT
`
`DATA
`
`DETERMINE
`
`LINK RATE
`
`
`
`HIGHER
`AUDIO/VIDEO
`LEVEL
`
`
` CHOOSE
`
`
`PAGE 9 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 9 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 9 of 11
`
`US 6,389,473 B1
`
`FIG. 7
`
`INPUT DATA
`
`
`
`CONTROL
`
`FROM 88
`
`
`
` CONTROL
`FROM 88
`
`82
`
`TO I-TTP 84
`
`PAGE 10 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 10 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 10 of 11
`
`US 6,389,473 B1
`
`F1(3.
`
`E3
`
`FROM
`SUCE 82
`
`84
`\
`
`
`
`94
`
`UNK DME—OUT FROM CHECK 88
`
`FTP SEND
`
`SUCE I
`
`TO UPDATE
`
`INDEX 86
`
`PAGE 11 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 11 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`U.S. Patent
`
`May 14, 2002
`
`Sheet 11 of 11
`
`US 6,389,473 B1
`
`88
`
`\
`
`FROM
`
`UPDATE 86
`
`FIG. 9
`
`
`
`
`CHECK SUCE
`TIME TSL
`
` RE—INITIALIZE A
`LINK
` COMPRESSION 9O
`
`
`
`DECREASE
`
`SLICE DURATION
`
`TO SET
`
`T DURATION 92
`
`YES
`
`E "uRc_REA§E" ' 7
`I SLICE DURATION I
`L. _ _ _ _ _ _ _ _J
`
`
`
`NEXT SLICE
`
`NO
`
`PAGE 12 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 12 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`US 6,389,473 B1
`
`1
`NETWORK MEDIA STREAMING
`
`A computer printout is attached hereto as an appendix in
`microfiche form and is incorporated herein by reference. The
`printout comprises executable program files in hexadecimal
`format. This appendix includes 2 microfiches, containing a
`total of 138 frames.
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to network data
`communications, and specifically to real-time multimedia
`broadcasting over a network.
`
`BACKGROUND OF THE INVENTION
`
`In network broadcasting, data are transmitted over a
`network in real time from a single transmitting computer to
`a plurality of clients simultaneously. The network may be a
`LAN, a WAN, an intranet or a public network such as the
`Internet. Network broadcasting is most commonly used to
`stream multimedia data, typically comprising images and
`sound.
`
`FIG. 1 is a schematic illustration showing a real-time
`broadcasting system 20, as is known in the art. One or more
`input devices 22 (for example, a video camera and/or
`microphone) are used to generate a multimedia data stream
`representing an entertainment or informational program to
`be transmitted to a plurality of clients 30 via a network 28.
`Because of bandwidth limitations of the network, the data
`stream from host 22 must first be compressed by a real-time
`encoder 24 and then routed to appropriate clients 30 by a
`broadcast server 26 (since not all clients on the network are
`necessarily intended to receive the broadcast).
`Encoder 24 and server 26 typically comprise high-cost,
`dedicated computer systems, such as a Sun Station
`(produced by Sun Microsystems) or a Windows NT server,
`running suitable RealSystem 5.0 software (produced by
`RealNetworks Inc., Seattle, Wash.). These dedicated sys-
`tems are required in order to ensure that the data stream is
`distributed and received by clients 30 in real time. Similarly,
`host 22 must typically be connected directly to encoder 24
`by a high-speed data link or LAN, and not via the Internet
`or other narrowband network. Therefore, real-time broad-
`casting is normally possible only for hosts having a suitable,
`dedicated encoder and broadcast server and cannot be
`
`offered by Internet service providers (ISPs) to their general
`clientele.
`
`SUMMARY OF THE INVENTION
`
`It is an object of some aspects of the present invention to
`provide substantially continuous, high-bandwidth data
`streaming over a network using common, existing server
`and network infrastructure.
`
`is a further object of some aspects of the present
`It
`invention to provide data broadcasting capability, particu-
`larly for multimedia data, without the need for a dedicated
`broadcast computer system.
`It
`is a further object of some aspects of the present
`invention to provide apparatus and methods for data broad-
`casting at reduced cost by comparison with systems known
`in the art.
`
`It is still another object of some aspects of the present
`invention to enable a personal computer to remotely broad-
`cast a multimedia program through an Internet service
`provider (ISP) using common, universally-supported Inter-
`net communication protocols.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`invention, a
`In preferred embodiments of the present
`transmitting computer generates a data stream and broad-
`casts the data stream via a network server to a plurality of
`clients. The data stream is divided into a sequence of
`segments or slices of the data, preferably time slices,
`wherein the data are preferably compressed. Each slice is
`preferably assigned a respective slice index. The transmit-
`ting computer uploads the sequence of slices to the server
`substantially in real
`time, preferably using an Internet
`protocol, most preferably the File Transfer Protocol (FTP),
`as is known in the art. The clients download the data stream
`
`from the server, preferably using an Internet protocol, as
`well, most preferably the Hypertext Transfer Protocol
`(HTTP), or alternatively, using other protocols, such as UDP
`or RTP, which are similarly known in the art. The clients use
`the slice indices of the frames to maintain proper synchro-
`nization of the playback. The division of the data stream into
`slices and the inclusion of the slice indices in the data stream
`
`to be used by the clients in maintaining synchronization
`allows the broadcast to go on substantially in real time
`without the use of special-purpose hardware.
`Preferably, each segment or slice is contained in a
`separate, respective file. Alternatively, the segments or slices
`may all be contained in a single indexed file, which is
`streamed to the client in a series of packets, each covering
`a range of one or more indices. HTTP version 1.1 supports
`this sort of file streaming. Other protocols may also be used
`for this purpose.
`In some preferred embodiments of the present invention,
`the data stream comprises multimedia data captured or
`generated by the transmitting computer. The term “multi-
`media” as used in the context of the present patent applica-
`tion and in the claims refers to images or sound or to data
`representative of images or of sound or a combination
`thereof. Multimedia image data may include still images,
`video, graphics, animation or any combination thereof,
`including text displayed in conjunction therewith. It will be
`appreciated, however,
`that
`the principles of the present
`invention may similarly be applied to streaming of other
`data types.
`the transmitting computer compresses the
`Preferably,
`frames in the data stream, most preferably using methods of
`image and audio compression such as those described in
`U.S. patent application Ser. No. 08/919,027, which is
`assigned to the assignee of the present patent application and
`incorporated herein by reference. Alternatively, any suitable
`methods of compression known in the art may be used. The
`compressed data are conveyed to the server and thence to the
`clients, which decompress the data.
`In some preferred embodiments of the present invention,
`the transmitting computer and the clients monitor the
`uploading and downloading of data to and from the server,
`respectively,
`in order to determine the amount of time
`required to convey each slice and to verify that the slices are
`conveyed at a sufficient rate. When the data stream com-
`prises multimedia data, the data rate should be generally
`equal to or faster than the rate at which the data are generated
`at the transmitting computer.
`In some of these preferred embodiments, the transmitting
`computer and/or the clients each open a plurality of FTP or
`HTTP links, respectively, with the network server. The slices
`are transferred over different ones of the links in alternation.
`
`Although typically none of the plurality of links has suffi-
`cient bandwidth on its own to convey the entire data stream
`in real time, the combined bandwidths of the plurality of
`links are generally sufficient for this purpose. Preferably,
`
`PAGE 13 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 13 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`US 6,389,473 B1
`
`3
`each of the links is monitored to determine its specific data
`transfer rate. If the transfer rate of any of the links is below
`a predetermined minimum, that link is preferably closed,
`and a new link is opened in its place.
`In other preferred embodiments, the slices are provided by
`the server at multiple resolution or quality levels. Each such
`level has a different degree of data compression, and thus
`corresponds to a different data bandwidth requirement. The
`client or the server monitors the data transfer rate of a data
`
`is
`that
`link opened therebetween and selects the level
`appropriate to the link bandwidth. If the monitored data
`transfer rate changes during transmission, the quality level is
`preferably reselected accordingly.
`Preferably, the transmitting computer monitors the band-
`width of the data stream that it is uploading to the server, and
`compares the data stream bandwidth to a known or esti-
`mated bandwidth of the link or links between the transmit-
`
`ting computer and the server. The transmitting computer
`preferably compresses the data stream at a compression ratio
`that is adjusted so as to match the data stream bandwidth to
`the available link bandwidth, using methods described, for
`example, in the above-mentioned U.S. patent application
`Ser. No. 08/919,027.
`There is therefore provided, in accordance with a pre-
`ferred embodiment of the present invention, a method for
`real-time broadcasting from a transmitting computer to one
`or more client computers over a network, including:
`providing at
`the transmitting computer a data stream
`having a given data rate;
`dividing the stream into a sequence of slices, each slice
`having a predetermined data size associated therewith;
`encoding the slices in a corresponding sequence of files,
`each file having a respective index; and
`uploading the sequence to a server at an upload rate
`generally equal to the data rate of the stream, such that
`the one or more client computers can download the
`sequence over the network from the server at a down-
`load rate generally equal to the data rate.
`Preferably, dividing the stream into the sequence of slices
`includes dividing the stream into a sequence of time slices,
`each having a predetermined duration associated therewith.
`Preferably, uploading the sequence includes comparing
`the upload rate to the data rate and adjusting the upload rate
`responsive to the comparison. Further preferably, encoding
`the stream includes compressing data in the stream at a
`desired compression ratio, and adjusting the upload rate
`includes changing the compression ratio. Alternatively or
`additionally, adjusting the upload rate includes adjusting the
`size of one or more of the slices.
`
`Preferably, uploading the sequence includes opening a
`plurality of file transfer links between the transmitting
`computer and the server, each link characterized by a
`respective link data rate, and uploading different files in the
`sequence over different ones of the plurality of links. Further
`preferably, opening the plurality of links includes opening
`links such that the data rates of the links taken together are
`sufficient to upload the sequence at the upload rate generally
`equal to the data rate.
`Preferably, uploading the sequence includes uploading a
`sequence using an Internet Protocol, most preferably using
`FTP.
`
`the method includes downloading the
`Preferably,
`sequence using an Internet protocol, most preferably HTTP,
`or alternatively, UDP or RTP, over the network from the
`server to the one or more client computers. Preferably, the
`one or more client computers decode the sequence and play
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`back the data stream responsive to the indices of the files, at
`a replay rate generally equal to the data rate.
`Preferably, uploading the sequence includes uploading
`and updating an index file containing the index of the file in
`the sequence that was most recently uploaded, and the one
`or more client computers read the index file to play back the
`sequence.
`In a preferred embodiment, downloading the
`sequence includes selecting a file in the sequence earlier
`than the file whose index is contained in the index file and
`
`least a portion of the sequence of files
`downloading at
`beginning with the selected file.
`Preferably, the one or more client computers include a
`plurality of client computers, and downloading the sequence
`includes downloading to the plurality of client computers
`substantially simultaneously.
`Preferably, downloading the sequence includes opening a
`plurality of download links between one of the client com-
`puters and the server, each link characterized by a respective
`link data rate, and downloading different
`files in the
`sequence over different ones of the plurality of links. Most
`preferably, opening the plurality of links includes opening
`links such that the data rates of the links taken together are
`sufficient to download the sequence at the download rate
`generally equal to the data rate.
`Preferably, opening the plurality of links includes moni-
`toring the data rates of the links and opening a new link in
`place of one of the links having a data rate lower than a
`predetermined level.
`In one preferred embodiment, opening the new link
`includes retransmitting at
`least one of the files in the
`sequence, wherein the at least one of the files was incom-
`pletely transmitted over the one of the links having the data
`rate lower than the predetermined level.
`In another preferred embodiment, opening the new link
`includes dropping at least one file out of the sequence,
`wherein the at
`least one of the files was incompletely
`transmitted over the one or more of the links having the data
`rate lower than the predetermined level.
`In still another preferred embodiment, encoding the slices
`includes encoding slices at a plurality of different quality
`levels, such that the files corresponding to a given one of the
`slices have a different, respective data size for each of the
`quality levels. Preferably, downloading the sequence
`includes determining a data bandwidth of the network
`between the server and the client computer and selecting one
`of the quality levels responsive to the determined band-
`width.
`
`Preferably, the data stream includes multimedia data.
`There is further provided, in accordance with a preferred
`embodiment of the present invention, apparatus for real-time
`broadcasting of a data stream having a given data rate over
`a network, including:
`a transmitting computer, which divides the stream into a
`sequence of slices, each slice having a predetermined
`data size associated therewith, and encodes the slices in
`a corresponding sequence of files, each file having a
`respective index, and
`which uploads the sequence to a server at an upload rate
`generally equal to the data rate, such that one or more
`client computers can download the sequence over the
`network from the server at a download rate generally
`equal to the data rate.
`Preferably,
`the transmitting computer compares the
`upload rate to the data rate and adjusts the upload rate
`responsive to the comparison. Most preferably, the trans-
`mitting computer compresses the data at a compression ratio
`which is varied responsive to the comparison. Additionally
`
`PAGE 14 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 14 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`US 6,389,473 B1
`
`5
`or alternatively, the transmitting computer adjusts the size of
`one or more of the slices responsive to the comparison.
`Preferably, the transmitting computer opens a plurality of
`links between the transmitting computer and the server, each
`link characterized by a respective data rate, and transmits
`different ones of the sequence of files over different ones of
`the plurality of links. Most preferably,
`the transmitting
`computer opens the plurality of links such that the data rates
`of the links taken together are sufficient
`to upload the
`sequence at the upload rate generally equal to the data rate.
`Further preferably, the transmitting computer monitors the
`data rates of the links and opens a new link in place of one
`of the links whose data rate is lower than a predetermined
`level.
`
`In a preferred embodiment, the slices are encoded at a
`plurality of different quality levels, such that the files cor-
`responding to a given one of the slices have a different,
`respective data size for each of the quality levels.
`Preferably,
`the transmitting computer uploads the
`sequence using an Internet upload protocol, most preferably
`FTP.
`
`Preferably, the one or more client computers decode the
`sequence and play back the data stream responsive to the
`indices thereof, at a data replay rate generally equal to the
`data rate. Preferably,
`the one or more client computers
`download the encode sequence using an Internet download
`protocol, most preferably HTTP or alternatively, UDP or
`RTP.
`
`Preferably, the one or more client computers include a
`plurality of client computers, which download the sequence
`substantially simultaneously.
`Preferably, the network includes the Internet.
`Further preferably, the data stream includes multimedia
`data, and the predetermined data size of each of the slices
`corresponds to a time duration of the slice.
`The present invention will be more fully understood from
`the following detailed description of the preferred embodi-
`ments thereof, taken together with the drawings in which:
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a schematic illustration of a computer broadcast
`network, as is known in the art;
`FIG. 2 is a schematic illustration of a computer broadcast
`network, in accordance with a preferred embodiment of the
`present invention;
`FIG. 3A is a block diagram that schematically illustrates
`a data structure of a broadcast sequence, in accordance with
`a preferred embodiment of the present invention;
`FIG. 3B is a block diagram that schematically illustrates
`an index file associated with the data structure of FIG. 3B,
`in accordance with a preferred embodiment of the present
`invention;
`FIG. 3C is a schematic illustration of a user interface
`
`graphic, for use in conjunction with the data structure of
`FIG. 3A, in accordance with a preferred embodiment of the
`present invention;
`FIG. 3D is a block diagram that schematically illustrates
`a data structure of a broadcast sequence, in accordance with
`another preferred embodiment of the present invention;
`FIG. 4 is a block diagram that schematically illustrates a
`network connection between a transmitting computer and a
`network server, for use in broadcasting of a data sequence,
`in accordance with a preferred embodiment of the present
`invention;
`FIG. 5 is a flow chart that schematically illustrates a
`method of uploading broadcast data from a transmitting
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`computer to a server, in accordance with a preferred embodi-
`ment of the present invention;
`FIG. 6A is a flow chart that schematically illustrates a
`method of downloading broadcast data from a server to a
`client, in accordance with a preferred embodiment of the
`present invention;
`FIG. 6B is a flow chart that schematically illustrates a
`method of downloading broadcast data from a server to a
`client, in accordance with another preferred embodiment of
`the present invention;
`FIG. 7 is a flow chart that schematically illustrates a
`method for preparing data files for transmission, in accor-
`dance with a preferred embodiment of the present invention;
`FIG. 8 is a flow chart that schematically illustrates a
`method of file transfer,
`in accordance with a preferred
`embodiment of the present invention; and
`FIG. 9 is a flow chart that schematically illustrates a
`method for monitoring network links, in accordance with a
`preferred embodiment of the present invention.
`DETAILED DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`Reference is now made to FIG. 2, which is a schematic
`illustration of a computer system 32 for remote broadcasting
`of a multimedia sequence over a network 28, in accordance
`with a preferred embodiment of the present
`invention.
`System 32 comprises a transmitting computer 34, which
`generates the sequence, a plurality of clients 30, and a
`network server 36, all of which communicate over network
`28, preferably using the well-known Internet Protocol (IP).
`Computer 34 preferably receives audiovisual input from
`input devices 22, although data inputs of other types may be
`generated at or by computer 34 using any suitable means
`known in the art.
`
`Network 28 preferably comprises the Internet, although it
`may equally comprise a LAN, WAN,
`intranet or other
`computer network as is known in the art. Computer 34 and
`clients 30 preferably comprise conventional personal com-
`puters or workstations. Server 36 may comprise any suitable
`type of computer or computer system, for example, a Sun
`Microsystems UltraSPARC station or a Windows NT server,
`as are commonly used by Internet Service Providers (ISPs)
`. In any case, it is noted that transmitting computer 34 can
`be remotely located relative to server 36, and that the server
`need not be equipped with any special-purpose hardware or
`software for real-time data broadcasting, unlike broadcast
`systems known in the art, such as the real-time encoder and
`broadcast server shown in FIG. 1.
`
`After preparing the multimedia sequence, computer 34
`uploads the sequence over network 28, preferably using the
`Internet File Transfer Protocol (FTP). Alternatively, other
`Internet protocols may be used, such as the TCP/IP, UDP or
`RT(X) protocols, which are known in the art. Preferably, the
`data in the sequence are compressed, although compression
`is not essential to implementation of the present invention.
`The sequence is preferably generated and compressed in real
`time, and could comprise, for example, an interview pro-
`gram or an entertainment or sports event, although a prere-
`corded sequence may similarly be broadcast in this manner.
`Computer 34 is preferably equipped with suitable software
`for preparing and compressing the multimedia sequence. For
`example, for audio data, the computer may typically run
`GSM 6.10 standard audio compression software, operating
`at a sample rate of 8 kHz, with 16 bits/sample. Some useful
`techniques for preparing, compressing and transmitting mul-
`timedia sequences are described in U.S. Pat. No. 5,841,432
`
`PAGE 15 OF 21
`
`|.M.L. SLU'S EXHIBIT 1003
`
`PAGE 15 OF 21
`
`I.M.L. SLU'S EXHIBIT 1003
`
`

`
`US 6,389,473 B1
`
`7
`and in the above-mentioned U.S. patent application Ser. No.
`08/919,027, both of which are incorporated herein by ref-
`erence.
`
`Clients 30 connect to server 36 and receive the multime-
`
`dia sequence, substantially in real time. Clients 30 prefer-
`ably download the sequence using the Hypertext Transfer
`Protocol (HTTP), although other Internet protocols may also
`be used, such as UDP or RTP, as noted hereinabove with
`reference to uploading by computer 34. Since FTP and
`HTTP are supported by substantially all network servers,
`server 36 need not include any special-purpose broadcasting
`hardware or software, as noted above. Similarly, because
`HTTP is supported by substantially all modern Web
`browsers, clients 30 will typically need only add a Java
`applet or plug-in to their existing Web browsers, as
`described further hereinbelow, in order to receive and play
`back the broadcast.
`
`FIG. 3A is a block diagram that schematically illustrates
`the structure of a stream of broadcast data 40 produced by
`computer 34, typically corresponding to a multimedia data
`sequence, in accordance with a preferred embodiment of the
`present invention. Data stream 40 comprises a series of data
`slices 42, 44, 46, 48, etc. Each slice contains a segment of
`video and/or audio data, corresponding to a respective,
`successive time interval labeled T1, T2, T3, etc. The data are
`preferably compressed, as described further hereinbelow.
`Computer 34 stores each slice as a corresponding file,
`having a running slice index 1, 2, 3 .
`.
`. N. Preferably, each
`file also includes one or more time stamps, indicating a real
`time at which the data in the file were recorded or an elapsed
`time relative to the beginning of stream 40. The files are
`uploaded to server 36, such that while any given slice (other
`than first slice 42) is being created, one or more preceding
`slices are in the process of being uploaded.
`Computer 34 monitors the time codes as file 40 is
`transmitted, and clients 30 similarly monitor the time codes
`as the file is received, in order to ensure that the transmission
`or reception is “keeping up” with the input of the data to the
`computer. In the event that a lag is detected, steps are taken
`to increase the data transmission or reception rate, as
`described further hereinbelow. For example, as shown in
`FIG. 3A, time intervals T1, T2, T3, etc., are not all equal, but
`rather are adjusted by computer 34 in response to the
`transmission rate. Alternatively or additionally, the compres-
`sion level of the data is varied, as is likewise described
`below, so as to adjust the data streaming rate to the available
`bandwidth over one or more channels between computer 34
`and server 36, and/or between server 36 and client 30.
`Computer 34 continues to upload files 42, 44, 46, etc.,
`until data stream 40 is finished or terminated by a user of
`computer 34. All of the files in the data stream may be saved
`on server 36 for any desired period of time, as long as the
`server has sufficient free memory that
`is accessible to
`computer 34. Typically, however, the memory available on
`server 36 is limited, and files 42, 44, 46, etc., will be stored
`on the server and erased therefrom in a “first-in-first-out”
`sequence.
`
`FIG. 3B is a block diagram that schematically illustrates
`an index file 50, which is created by computer 34, and is
`uploaded to server 36,
`in accordance with a preferred
`embodiment of the present invention. The index file com-
`prises a slice ID 52, indicating the index of the file in data
`stream 40 that was most recently uploaded by computer 34.
`Each time a new file 42, 44, 46, etc., is uploaded, ID 52 in
`file 50 on server 36 is updated. Preferably, ID 52 holds the
`file name of the new file, wherein the name typically
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`comprises a string followed by the index of the file. When
`one of computers 30 connects to server 36 and begins to
`download the data stream, it first reads the index file in order
`to identify at what point in stream 40 to begin and to start
`receiving the data stream substantially in real time, prefer-
`ably with only a minimal lag, as it is transmitted from
`computer 34. Alternatively, a user of one of computers 30
`may choose to begin downloading data stream 40 from an
`earlier point in time than that indicated by ID 52. Further
`alternatively, stream 40 may be multicast to clients 30, as is
`known in the art, typically without the use of an index file.
`Index file 50 may further include a message 54, which is
`read by computers 30 when they connect to server 36 to
`download data stream 40 or, alternatively or additionally, at
`any time the message is updated by computer 34. The
`message contains parameters relating generally to the data
`stream and/or instructions to computers 30, for example,
`“transmission paused.” FIG. 3C is a schematic representa-
`tion of a user interface graphic “slider” 55, available to users
`of computers 30, in accordance with a preferred embodiment
`of the present invention. Slider 55, which is preferably
`displayed on the screens of computers 30, includes a bar 56
`and a movable indicator 58. The symbols J, J+1, J+2, .
`.
`. N
`in the figure are the indices of the slices of stream 40 that are
`stored on server 36, wherein N is the index of the most
`recent slice, and J is the index of the earliest stored slice. J
`may indicate the first slice in the sequence, if all of the files
`are stored on server 36, or it may be the earliest file not yet
`erased. (The indices are marked in the figure on bar 56 for
`clarity, and need not actually be shown on the computer
`screen.)
`When one of computers 30 reads index file 50 and begins
`to download stream 40, indicator 58 preferably marks the
`most recent slice, as shown in FIG. 3C. This is the point at
`which the download will begin, unless the user of the
`computer chooses otherwise. If the user wishes to begin the
`download at an earlier point, he may move indicator 58 to
`the left along bar 56 to that point, preferably using a mouse
`or other pointing device, as is known in the art. Indicator 58
`may be moved back and forth along bar 56 to jump back and
`forth along stream 40.
`FIG. 3D is a block diagram that schematically illustrates
`a file format of a multi-level data stream 41, in accordance
`with another preferred embodiment of the present invention.
`The data stre