US006389473B1
`a2) United States Patent (10) Patent No:—US 6,389,473 B1
`
`Carmeletal.
`(45) Date of Patent:
`May14, 2002
`
`
`(54) NETWORK MEDIA STREAMING
`
`(75)
`
`Inventors: Sharon Carmel; Tzur Daboosh, both
`of Giv’atayim; Eli Reifman, Rishon le
`Zion; Naftali Shani; Ziv Eliraz, both
`of Tel Aviv; Dror Ginsberg, Karkur;
`Edan Ayal, Ktar Saba,all of (IL)
`(73) Assignee: Geo Interactive Media Group Ltd.,
`Givatayim (IL)
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/275,703
`(22) Filed:
`Mar.24, 1999
`(30)
`Foreign Application Priority Data
`Mar. 24, 1998
`IL) seeseeceeeee cee ceeesceneeeneeaeeaes 123819
`(51)
`Int. Ch.ee necreeeeees GO06F 13/00
`(52) US. CD. cece cessesesesenensneteceescsesenenens 709/231
`(58) Field of Search oo... 707/500.1; 709/200,
`709/231, 236, 246, 247: 382/236, 239:
`375/240.12
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,267,334 A
`11/1993 Normille et al. 0... 382/236
`
`5,404,446 A
`5,841,432 A
`
`4/1995. Bowater et al. vcsscssse 345/537
`11/1998 Carmel et al... 707/500.1
`
`Primary Examiner—Robert B. Harrell
`(74) Attorney, Agent, or Firm—Ladas & Parry
`
`(57)
`
`ABSTRACT
`
`A method for real-time broadcasting from a transmitting
`computer to one or more client computers over a network,
`.
`.
`.
`_.
`including providing at
`the transmitting computer a data
`gag
`g
`stream having a
`given data rate, and dividing the stream into
`a sequenceofslices, each slice having a predetermined data
`size associated therewith. The slices are encoded in a
`corresponding sequence of files, each file having a respec-
`tive index, and the sequence is uploaded to a server at an
`upload rate generally cqual to the data rate of the strcam,
`such that the one or moreclient computers can download the
`sequence over the network from the server at a download
`rate generally equal to the data rate.
`
`41 Claims, 11 Drawing Sheets
`
`Microfiche Appendix Included
`(2 Microfiche, 138 Pages)
`
`NETWORK
`SERVER
`
`28
`|
`
`+=)
`—
`
`28
`
` STANDARD
`
`
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0001
`EXHIBIT 1004 - PAGE 0001
`
`

`

`U.S. Patent
`
`May14, 2002
`
`Sheet 1 of UL
`
`US 6,389,473 B1
`
`
`
`om”
`
`s
`a>
`S&
`
`— 2O
`
`REALTIME
`
`ENCODE
`
`26
`
`24
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0002
`EXHIBIT 1004 - PAGE 0002
`
`©a
`
`N
`
`r
`
`ea
`<=
`e
`ox
`Os—- x
`tn a
`
`A/ey]
`
`w/”Ns
`
`

`

`U.S. Patent
`
`May14, 2002
`
`Sheet 2 of LL
`
`US 6,389,473 B1
`
`N o
`
`O
`Le
`
`
`
`
`
`9
`
`|
`
`2 I
`
`t“
`
`NETWORK
`
`STANDARD
`
`SERVER
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0003
`EXHIBIT 1004 - PAGE 0003
`
`

`

`U.S. Patent
`
`May 14, 2002
`
`Sheet 3 of I
`
`US 6,389,473 B1
`
`mp FIG. 3A
`
`
`
`0”
`
`TIME
`
`FIG. 3B
`
`SLICE N
`
`52
`
`
`MESSAGE
`
`FIG. 3C
`
`
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0004
`EXHIBIT 1004 - PAGE 0004
`
`

`

`6S
`
`19
`
`LS
`
`U.S. Patent
`
`qd¢“Old
`
`Ly
`
`LS
`
`by
`
`LvGY
`
`Lv
`
`GP
`
`cHGAS)
`
`Olan
`
`(Z)
`
`z#BAN
`
`olan
`
`(1)
`
`6SZS cv
`
`May14, 2002
`
`Sheet 4 of 11
`
`US 6,389,473 B1
`
`Sid
`
`ZHOncH4OAGIA
`olany
`
`L#GAH]
`
`olan
`
`(v)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0005
`EXHIBIT 1004 - PAGE 0005
`
`
`

`

`U.S. Patent
`
`May 14, 2002
`
`Sheet 5 of I
`
`US 6,389,473 B1
`
`4 F
`
`IG.
`
`oO
`ro
`
`xe
`oe
`oS
`= &WwW
`WA
`
`°Q
`oS
`
`nN
`Ta)
`
`Te)
`oo
`
`Oo
`oe]
`ao, &
`
`— Nf]
`x
`x
`x
`=
`z=
`=
`md
`—_
`—_!
`
`st]
`xf
`=|
`—
`
`jo]
`jxe|
`j=,
`woed
`
`[wo
`ix
`/=
`—_
`
`
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0006
`EXHIBIT 1004 - PAGE 0006
`
`

`

`U.S. Patent
`
`May14, 2002
`
`Sheet 6 of 11
`
`US 6,389,473 B1
`
`FIG. S
`
`
`
`CONNECT
`
`
` INPUT
`
`BROACAST
`
`
`TO SERVER
`DATA
`
` ENCODE
`
`80 82
`FTP TO SERVER 84
` 86
` 88
` CHECK
`
`
`UPDATE
`INDEX
`FILE
`
`LINK
`FUNCTION
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0007
`EXHIBIT 1004 - PAGE 0007
`
`

`

`U.S. Patent
`
`May 14, 2002
`
`Sheet 7 of I
`
`US 6,389,473 B1
`
`FIG. 6A
`
`CONNECT
`TO SERVER
`
`HTTP FROM
`SERVER
`
`READ INDEX
`FILE
`
`
`
`
`
`
`
`
`
`
`
`
`CONTINUE
`
`
`
`CONNECT
`NEW LINK
`
`SELECT SLICE
`
`DECODE
`
`OUTPUT
`DATA
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0008
`EXHIBIT 1004 - PAGE 0008
`
`

`

`U.S. Patent
`
`May 14, 2002
`
`Sheet 8 of I
`
`US 6,389,473 B1
`
`FIG. 6B
`
`CONTINUE
`
`
`
`
`CONNECT
`TO SERVER
`
`HTTP FROM
`SERVER
`
`READ HEADER
`
`CHOOSE
`LEVEL
`
`READ SLICE
`
`DECODE
`
`OUTPUT
`DATA
`
`DETERMINE
`LINK RATE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`CHOOSE
`LOWER
`
`AUDIO/VIDEO
`
`LEVEL
`
`
`
` CHOOSE
`RATE
`HIGHER
`
`HIGHER THAN
`
`
`AUDIO/VIDEO
`
`
`
`NEED
`
`LEVEL
`?
`
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0009
`EXHIBIT 1004 - PAGE 0009
`
`

`

`U.S. Patent
`
`May 14, 2002
`
`Sheet 9 of I
`
`US 6,389,473 B1
`
`FIG. 7
`
`80
`
`82
`
`INPUT DATA
`
`SET COMPRESSION
`
`COMPRESS DATA
`
`PREPARE
`SLICE I
`
`TQ FIP 84
`
`RATIO
`
`CONTROL
`FROM 88
`
`
`
`
`C
`CONTROL
`
`
`SET SLICE DURATION
`FROM 88
`
`
`
`
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0010
`EXHIBIT 1004 - PAGE 0010
`
`

`

`U.S. Patent
`
`May14, 2002
`
`Sheet 10 of 11
`
`US 6,389,473 B1
`
`FIG. 8
`
`84
`
`FROM
`SLICE 82
`
`LINK TIME-OUT FROM CHECK 88
`
`\
`
`FTP OPEN
`LINK J
`
`
`
` 94
`
`FTP SEND
`SUCE I
`
`TO UPDATE
`INDEX 86
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0011
`EXHIBIT 1004 - PAGE 0011
`
`

`

`U.S. Patent
`
`May14, 2002
`
`Sheet 11 of
`
`US 6,389,473 B1
`
`88\
`
`ROM
`
`FIG. 9
`
`UPDATE 86
`
`CHECK SLICE
`
`RE-INITIALIZE {:
`LINK
`
`OPEN 94
`
`INCREASE
`COMPRESSION
`
`COMPRESSION 90
`
`DECREASE
`SLICE DURATION
`
`TIME Tsp
`
`
`
`
`TO SET
`DURATION 92
`
`
` “INCREASE |
`
`
`| SLICE DURATION |
`
`Wwe eee —_
`
`~
`
`80 NEXT SLICE
`
`TO ENCODE
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0012
`EXHIBIT 1004 - PAGE 0012
`
`

`

`US 6,389,473 Bl
`
`1
`NETWORK MEDIA STREAMING
`
`A computer printout is attached hereto as an appendix in
`microfiche form andis incorporated herein byreference. The
`printout comprises executable program files in hexadecimal
`format. This appendix includes 2 microfiches, containing a
`total of 138 frames.
`
`FIELD OF THE INVENTION
`
`‘The present invention relates generally to network data
`communications, and specifically to real-time multimedia
`broadcasting over a network.
`
`BACKGROUND OF THE INVENTION
`
`In network broadcasting, data are transmitted over a
`network in real time from a single transmitting computer to
`a plurality of clicnts simultancously. The nctwork may be a
`LAN, a WAN,an intranet or a public network such as the
`Internet. Network broadcasting is most commonlyused to
`stream multimedia data, typically comprising images and
`sound.
`
`FIG. 1 is a schematic illustration showing a real-time
`broadcasting system 290, as is knowninthe art. One or more
`input devices 22 (for example, a video camera and/or
`microphone) are used to generate a multimedia data stream
`representing an cntcrtainment or informational program to
`be transmitted to a plurality of clients 30 via a network 28.
`Because of bandwidth limitations of the network, the data
`stream from host 22 must first be compressed bya real-time
`encoder 24 and then routed to appropriate clients 30 by a
`broadcast server 26 (since notall clients on the network are
`necessarily intended to receive the broadcast).
`Encoder 24 and server 26 typically comprise high-cost,
`dedicated computer systems, such as a Sun Station
`(produced by Sun Microsystems) or a Windows NTserver,
`running suitable RealSystem 5.0 software (produced by
`RealNetworks Inc., Seattle, Wash.). These dedicated sys-
`tems are required in order to ensure that the data stream is
`distributed and received by clients 30 in real time. Similarly,
`host 22 must typically be connected directly to encoder 24
`by a high-speed data link or LAN,and not via the Internet
`or olher narrowband network. Therefore, real-time broad-
`casting is normally possible only for hosts having a suitable,
`dedicated encoder and broadcast server and cannot be
`offered by Internet service providers (ISPs) to their general
`clientele.
`
`SUMMARYOF THE INVENTION
`
`It is an object of some aspects of the present invention to
`provide substantially continuous, high-bandwidth data
`streaming over a network using common, existing server
`and network infrastructure.
`
`is a further object of some aspects of the present
`It
`invention to provide data broadcasting capability, particu-
`larly for multimedia data, without the need for a dedicated
`broadcast computer system.
`It
`is a further object of some aspects of the present
`invention to provide apparatus and methodsfor data broad-
`casting at reduced cost by comparison with systems known
`in theart.
`
`It is still another object of some aspects of the present
`invention to enable a personal computer to remotely broad-
`cast a multimedia program through an Internet service
`provider (ISP) using common,universally-supported Inter-
`net communication protocols.
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`invention, a
`In preferred embodiments of the present
`transmitting computer generates a data stream and broad-
`casts the data stream via a network server to a plurality of
`clients. The data stream is divided into a sequence of
`segments or slices of the data, preferably time slices,
`wherein the data are preferably compressed. Eachslice is
`preferably assigned a respective slice index. The transmit-
`ting computer uploads the sequence of slices to the server
`substantially in real
`time, preferably using an Internet
`protocol, most preferably the File Transfer Protocol (FTP),
`as is knownin the art. The clients download the data stream
`
`from the server, preferably using an Internet protocol, as
`well, most preferably the Hypertext Transfer Protocol
`(HTTP), oralternatively, using other protocols, such as UDP
`or RTP, which are similarly knownin the art. The clients use
`the slice indices of the frames to maintain proper synchro-
`nization of the playback. The division of the data stream into
`slices and the inclusion oftheslice indicesin the data stream
`to be used by the clients in maintaining synchronization
`allows the broadcast to go on substantially in real
`time
`without the use of special-purpose hardware.
`Preferably, each segment or slice is contained in a
`separate, respectivefile. Alternatively, the segmentsor slices
`may all be contained in a single indexed file, which is
`streamed to the client in a series of packets, each covering
`a range of one or more indices. HTTP version 1.1 supports
`this sort of file streaming. Other protocols may also be used
`for this purpose.
`In some preferred embodiments of the present invention,
`the data stream comprises multimedia data captured or
`generated by the transmitting computer. The term “multi-
`media” as used in the context of the present patent applica-
`tion and in the claims refers to images or sound or to data
`representative of images or of sound or a combination
`thereof. Multimedia image data may include still images,
`video, graphics, animation or any combination thereof,
`including text displayed in conjunction therewith. It will be
`appreciated, however,
`that
`the principles of the present
`invention may similarly be applied to streaming of other
`data types.
`the transmitting computer compresses the
`Prefcrably,
`framesin the data stream, most preferably using methods of
`image and audio compression such as those described in
`U.S. patent application Ser. No. 08/919,027, which is
`assigned to the assignee of the presentpatent application and
`incorporated herein by reference. Alternatively, any suitable
`methods of compression knownin the art may be used. The
`compressed data are conveyed to the server and thence to the
`clients, which decompress the data.
`In somepreferred embodiments of the present invention,
`the transmitting computer and the clients monitor the
`uploading and downloading of data to and from the server,
`respectively,
`in order to determine the amount of time
`required to convey eachslice andto verify that the slices are
`conveyed at a sufficient rate. When the data stream com-
`prises multimedia data, the data rate should be generally
`equal to or faster than the rate at which the data are generated
`at the transmitting computer.
`In someof these preferred embodiments,the transmitting
`computer and/or the clients each open a plurality of FTP or
`HTTPlinks, respectively, with the network server. The slices
`are transferred over different onesof the links in alternation.
`
`Although typically none of the plurality of links has suffi-
`cient bandwidth on its own to convey the entire data stream
`in real time, the combined bandwidths of the plurality of
`links are generally sufficient for this purpose. Preferably,
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0013
`EXHIBIT 1004 - PAGE 0013
`
`

`

`US 6,389,473 Bl
`
`3
`cach ofthe links is monitored to determine its specific data
`transfer rate. If the transfer rate of any of the links is below
`a predetermined minimum, that link is preferably closed,
`and a new link is openedinits place.
`In other preferred embodiments,the slices are provided by
`the server at multiple resolution or quality levels. Each such
`level has a different degree of data compression, and thus
`correspondsto a different data bandwidth requirement. The
`client or the server monitors the data transfer rate of a data
`
`is
`that
`link opened therebetween and selects the level
`appropriate to the link bandwidth. If the monitored data
`transfer rate changes during transmission, the quality levelis
`preferably reselected accordingly.
`Preferably, the transmitting computer monitors the band-
`width of the data stream that it is uploading to the server, and
`compares the data stream bandwidth to a known oresti-
`mated bandwidth of the link or links between the transmit-
`
`ting computer and the server. The transmitting computer
`preferably compressesthe data stream at a compression ratio
`that is adjusted so as to match the data stream bandwidth to
`the available link bandwidth, using methods described, for
`example, in the above-mentioned U.S. patent application
`Ser. No. 08/919,027.
`There is therefore provided, in accordance with a pre-
`ferred embodiment of the present invention, a method for
`real-time broadcasting from a transmitting computer to one
`or more client computers over a network, including:
`providing at
`the transmitting computer a data stream
`having a given data rate;
`dividing the stream into a sequence ofslices, each slice
`having, a predetermined data size associated therewith;
`encoding the slices in a corresponding sequenceoffiles,
`each file having a respective index; and
`uploading the sequence to a server at an upload rate
`generally equal to the data rate of the stream, such that
`the one or more clicnt computers can download the
`sequence over the network from the server at a down-
`load rate generally equal to the data rate.
`Preferably, dividing the stream into the sequenceofslices
`includes dividing the stream into a sequence of timeslices,
`each having a predetermined duration associated therewith.
`Preferably, uploading the sequence includes comparing
`the upload rate to the data rate and adjusting the uploadrate
`responsive to the comparison. Further preferably, encoding
`the stream includes compressing data in the stream at a
`desired compression ratio, and adjusting the upload rate
`includes changing the compression ratio. Alternatively or
`additionally, adjusting the upload rate includes adjusting the
`size of one or more ofthe slices.
`
`Preferably, uploading the sequence includes opening a
`plurality of file transfer links between the transmitting
`computer and the server, each link characterized by a
`respective link data rate, and uploading differentfiles in the
`sequence overdifferent ones ofthe plurality of links. Further
`preferably, opening the plurality of links includes opening
`links such that the data rates of the links taken together are
`sufficient to upload the sequence at the upload rate generally
`equal to the data rate.
`Preferably, uploading the sequence includes uploading a
`sequence using an Internet Protocol, most preferably using
`FTP.
`the method includes downloading the
`Preferably,
`sequence using an Internet protocol, most preferably HTTP,
`or alternatively, UDP or RTP, over the network from the
`server to the one or more client computers. Preferably, the
`one or more client computers decode the sequence and play
`
`4
`back the data stream responsive to the indices ofthefiles, at
`a replay rate generally equal to the data rate.
`Preferably, uploading the sequence includes uploading
`and updating an indexfile containing the index ofthe file in
`the sequence that was most recently uploaded, and the one
`or more client computers read the index file to play back the
`sequence.
`In a preferred embodiment, downloading the
`sequence includes selecting a file in the sequence earlier
`than the file whose index is contained in the index file and
`
`least a portion of the sequence of files
`downloading at
`beginning with the selectedfile.
`Preferably, the one or more client computers include a
`plurality of client computers, and downloading the sequence
`includes downloading to the plurality of client computers
`substantially simultaneously.
`Preferably, downloading the sequence includes opening a
`plurality of download links between one of the client com-
`puters and the server, each link characterized by a respective
`link data rate, and downloading different
`files in the
`sequence overdifferent ones of the plurality of links. Most
`preferably, opening the plurality of links includes opening
`links such that the data rates of the links taken together are
`sufficient to download the sequence at the download rate
`generally equal to the data rate.
`Preferably, opening the plurality of links includes moni-
`toring the data rates of the links and opening a newlink in
`place of one of the links having a data rate lower than a
`predetermined level.
`In one preferred embodiment, opening the new link
`includes retransmitting at
`least one of the files in the
`sequence, wherein the at least one of the files was incom-
`pletely transmitted over the one of the links having the data
`rate lower than the predetermined level.
`In another preferred embodiment, opening the new link
`includes dropping, at least one file out of the sequence,
`wherein the at
`least one of the files was incompletely
`transmitted over the one or moreofthe links having the data
`rate lower than the predetermined level.
`In still another preferred embodiment, encodingthe slices
`includes encoding slices at a plurality of different quality
`levels, such that the files corresponding to a given one of the
`slices have a different, respective data size for each of the
`quality levels. Preferably, downloading the sequence
`includes determining a data bandwidth of the network
`between the server and the client computer and selecting one
`of the quality levels responsive to the determined band-
`width.
`
`Preferably, the data stream includes multimedia data.
`‘There is further provided, in accordance with a preferred
`embodimentof the present invention, apparatus for real-time
`broadcasting of a data stream having a given data rate over
`a network, including:
`a transmitting computer, which divides the stream into a
`sequence of slices, each slice having a predetermined
`data size associated therewith, and encodesthe slices in
`a corresponding sequence offiles, each file having a
`respective index, and
`which uploads the sequence to a server at an upload rate
`generally equal to the data rate, such that one or more
`client computers can download the sequence over the
`network from the server at a download rate generally
`equal to the datarate.
`Preferably,
`the transmitting computer compares the
`upload rate to the data rate and adjusts the upload rate
`responsive to the comparison. Most preferably, the trans-
`mitting computer compresses the data at a compressionratio
`whichis varied responsive to the comparison. Additionally
`
`10
`
`30
`
`35
`
`40
`
`45
`
`50
`
`60
`
`65
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0014
`EXHIBIT 1004 - PAGE 0014
`
`

`

`US 6,389,473 Bl
`
`5
`or alternatively, the transmitting computer adjusts the size of
`one or more of the slices responsive to the comparison.
`Preferably, the transmitting computer opensa plurality of
`links between the transmitting computer and the server, each
`link characterized by a respective data rate, and transmits
`different ones of the sequenceoffiles over different ones of
`the plurality of links. Most preferably,
`the transmitting
`computer opensthe plurality of links such that the data rates
`of the links taken together are sufficient
`to upload the
`sequence at the upload rate generally equal to the data rate.
`Further preferably, the transmitting computer monitors the
`data rates of the links and opens a new link in place of one
`of the links whose data rate is lower than a predetermined
`level.
`
`In a preferred embodiment, the slices are encoded at a
`plurality of different quality levels, such that the files cor-
`responding to a given one of the slices have a different,
`respective data size for each of the quality levels.
`Preferably,
`the transmitting computer uploads the
`sequence using an Internet upload protocol, most preferably
`FTP.
`
`Preferably, the one or more client computers decode the
`sequence and play back the data stream responsive to the
`indices thereof, at a data replay rate generally equal to the
`data rate. Preferably,
`the one or more client computers
`download the encode sequence using an Internet download
`protocol, most preferably HTTP or alternatively, UDP or
`RTP.
`
`Preferably, the one or more client computers include a
`plurality of client computers, which download the sequence
`substantially simultaneously.
`Preferably, the network includes the Internet.
`Further preferably, the data stream includes multimedia
`data, and the predetermined data size of each of the slices
`corresponds to a time duration of theslice.
`The present invention will be more fully understood from
`the following detailed description of the preferred embodi-
`ments thereof, taken together with the drawings in which:
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG.1 is a schematicillustration of a computer broadcast
`network, as is known in the art;
`FIG.2 is a schematic illustration of a computer broadcast
`network, in accordance with a preferred embodimentof the
`present invention;
`FIG. 3A is a block diagram that schematicallyillustrates
`a data structure of a broadcast sequence, in accordance with
`a preferred embodiment of the present invention;
`FIG.3B is a block diagram that schematicallyillustrates
`an index file associated with the data structure of FIG. 3B,
`in accordance with a preferred embodiment of the present
`invention;
`FIG. 3C is a schematic illustration of a user interface
`
`graphic, for use in conjunction with the data structure of
`FIG. 3A, in accordance with a preferred embodimentof the
`present invention;
`FIG. 3D is a block diagramthat schematically illustrates
`a data structure of a broadcast sequence, in accordance with
`another preferred embodiment of the present invention;
`FIG.4 is a block diagram that schematically illustrates a
`network connection between a transmitting computer and a
`network server, for use in broadcasting of a data sequence,
`io accordance with a preferred embodiment of the present
`invention;
`FIG. 5 is a flow chart that schematically illustrates a
`method of uploading broadcast data [rom a transmitting
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`computer to a server, in accordance with a preferred embodi-
`ment of the present invention;
`FIG. 6A is a flow chart that schematically illustrates a
`method of downloading broadcast data from a server to a
`client, in accordance with a preferred embodimentof the
`present invention;
`FIG. 6B is a flow chart that schematically illustrates a
`method of downloading broadcast data from a server to a
`client, in accordance with another preferred embodiment of
`the present invention;
`that schematically illustrates a
`FIG. 7 is a flow chart
`method for preparing data files for transmission, in accor-
`dance with a preferred embodimentofthe present invention;
`FIG. 8 is a flow chart
`that schematically illustrates a
`method of file transfer,
`in accordance with a preferred
`embodiment of the present invention; and
`FIG. 9 is a flow chart
`that schematically illustrates a
`method for monitoring network links, in accordance with a
`preferred embodiment of the present invention.
`DETAILED DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`Reference is now madeto FIG. 2, which is a schematic
`illustration of a computer system 32 for remote broadcasting
`of a multimedia sequence over a network 28, in accordance
`with a preferred embodiment of the present
`invention.
`System 32 comprises a transmitting computer 34, which
`generates the sequence, a plurality of clients 30, and a
`networkserver 36, all of which communicate over network
`28, preferably using the well-known Internet Protocol(IP).
`Computer 34 preferably receives audiovisual input from
`input devices 22, although data inputs of other types may be
`generated at or by computer 34 using any suitable means
`known in theart.
`
`Network 28 preferably comprises the Internet, althoughit
`may equally comprise a LAN, WAN,
`intranet or other
`computer network as is known in the art. Computer 34 and
`clients 30 preferably comprise conventional personal com-
`puters or workstations. Server 36 may comprise any suitable
`type of computer or computer system, for example, a Sun
`Microsystems UltraSPARCstation or a WindowsNTserver,
`as are commonly used by Internet Service Providers (ISPs)
`. In anycasc,it is noted that transmitting computer 34 can
`be remotely located relative to server 36, and that the server
`need not be equipped with any special-purpose hardware or
`software for real-time data broadcasting, unlike broadcast
`systems knownin theart, such as the real-time encoder and
`broadcast server shown in FIG. 1.
`
`After preparing the multimedia sequence, computer 34
`uploads the sequence over network 28, preferably using the
`Internet File Transfer Protocol (FTP). Alternatively, other
`Internet protocols maybe used, such as the TCP/IP, UDPor
`RT(x) protocols, which are knownin theart. Preferably, the
`data in the sequence are compressed, although compression
`is not essential to implementation of the present invention.
`The sequenceis preferably generated and compressedin real
`time, and could comprise, for example, an interview pro-
`gram or an entertainment or sports event, although a prere-
`corded sequence may similarly be broadcast in this manner.
`Computer 34 is preferably equipped with suitable software
`for preparing and compressing the multimedia sequence. For
`example, for audio data, the computer may typically run
`GSM 6.10 standard audio compression software, operating
`at a sample rate of 8 kHz, with 16 bits/sample. Some useful
`techniques for preparing, compressing and transmitting mul-
`timedia sequences are described in U.S. Pat. No. 5,841,432
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1004 - PAGE 0015
`EXHIBIT 1004 - PAGE 0015
`
`

`

`US 6,389,473 Bl
`
`7
`and in the above-mentioned U.S. patent application Scr. No.
`08/919,027, both of which are incorporated herein by ref-
`erence.
`
`Clients 30 connect to server 36 and receive the multime-
`dia sequence, substantially in real time. Clients 30 prefer-
`ably download the sequence using the Hypertext Transfer
`Protocol (HTTP), although other Internet protocols may also
`be used, such as UDP or RIP, as noted hereinabove with
`reference to uploading by computer 34. Since FTP and
`HTTPare supported by substantially all network scrvers,
`server 36 need not include anyspecial-purpose broadcasting
`hardware or software, as noted above. Similarly, because
`HTTP is supported by substantially all modern Web
`browsers, clients 30 will typically need only add a Java
`applet or plug-in to their existing Web browsers, as
`described further hereinbelow, in order to receive and play
`back the broadcast.
`
`
`
`FIG. 3A is a block diagram that schematicallyillustrates
`he structure of a stream of broadcast data 40 produced by
`computer 34, typically corresponding to a multimedia data
`sequence, in accordance with a preferred embodimentof the
`present invention. Data stream 40 comprises a series of data
`slices 42, 44, 46, 48, etc. Each slice contains a segment of
`video and/or audio data, corresponding to a respective,
`successive time interval labeled T,, T,, T;, etc. The data are
`preferably compressed, as described further hereinbelow.
`Computer 34 stores each slice as a corresponding file,
`having a running slice index 1,2,3...N. Preferably, each
`file also includes one or more time stamps, indicating a real
`ime at which the data in the file were recorded or an elapsed
`ime relative to the beginning of stream 40. The files are
`uploaded to server 36, such that while any given slice (other
`hanfirst slice 42) is being created, one or more preceding
`slices are in the process of being uploaded.
`Computer 34 monitors the time codes as file 40 is
`ransmitted, and clients 30 similarly monitor the time codes
`as the file is received, in order to ensure that the transmission
`or receptionis “keeping up” with the input of the data to the
`computer. In the event that a lag is detected, steps are taken
`to increase the data transmission or reception rate, as
`described further hercinbelow. For example, as shown in
`FIG. 3A,time intervals T,, T., T,, etc., are not all equal, but
`rather are adjusted by computer 34 in response to the
`transmissionrate. Alternatively or additionally, the compres-
`sion level of the data is varied, as is likewise described
`below, so as to adjust the data streamingrate to the available
`bandwidth over one or more channels between computer 34
`and server 36, and/or between server 36 and client 30.
`Computer 34 continues to upload files 42, 44, 46, etc.,
`until data stream 40 is finished or terminated by a user of
`computer 34. All of the files in the data stream may be saved
`on server 36 for any desired period of time, as long as the
`server has sufficient free memory that
`is accessible to
`computer 34. Typically, however, the memory available on
`server 36 is limited, and files 42, 44, 46, etc., will be stored
`on the server and erased therefrom in a “first-in-first-out”
`sequence.
`
`FIG. 3B is a block diagram that schematicallyillustrates
`an index file 50, which is created by computer 34, and is
`uploaded to server 36,
`in accordance with a preferred
`embodiment of the present invention. The index file com-
`prises a slice ID 52, indicating the index of the file in data
`stream 40 that was most recently uploaded by computer 34.
`Each time a new file 42, 44, 46, etc., is uploaded, ID 52 in
`file 50 on server 36 is updated. Preferably, ID 52 holds the
`file name of the new file, wherein the name typically
`
`8
`comprises a string followed by the index of the file. When
`one of computers 30 connects to server 36 and begins to
`download the data stream,it first reads the index file in order
`to identify at what point in stream 40 to begin andto start
`receiving the data stream substantially in real time, prefer-
`ably with only a minimal lag, as it is transmitted from
`computer 34. Alternatively, a user of one of computers 30
`may choose to begin downloading data stream 40 from an
`earlier point in time than that indicated by ID 52. Further
`alternatively, stream 40 may be multicast to clients 30, as is
`knownin the art, typically without the use of an indexfile.
`Index file 50 may further include a message 54, which is
`read by computers 30 when they connect to server 36 to
`download data stream 40 or, alternatively or additionally, at
`any time the message is updated by computer 34. The
`message contains parameters relating generally to the data
`stream and/or instructions to computers 30, for example,
`“transmission paused.” FIG. 3C is a schematic representa-
`tion of a user interface graphic “slider” 55, available to users
`of computers 30, in accordance with a preferred embodiment
`of the present invention. Slider 55, which is preferably
`displayed on the screens of computers 30, includes a bar 56
`and a movable indicator 58. The symbols J, J+1, J+2,...N
`in the figure are the indices of the slices of stream 40 that are
`stored on server 36, wherein N is the index of the most
`recent slice, and J is the index of the earliest stored slice. J
`may indicate thefirst slice in the sequence,if all of the files
`are stored on server 36, or it may bethe earliest file not yet
`erased. (The indices are marked in the figure on bar 56 for
`clarity, and need not actually be shown on the computer
`screen.)
`Whenone of computers 30 reads indexfile 50 and begins
`to download stream 40, indicator 58 preferably marks the
`most recent slice, as shown in FIG. 3C. This is the point at
`which the downoload will begin, uoless the user of the
`computer chooses otherwise. If the user wishes to begin the
`download at an earlier point, he may move indicator 58 to
`the left along bar 56 to that point, preferably using a mouse
`or other pointing device, as is knownin theart. Indicator 58
`may be movedbackand forth along bar 56 to jump back and
`forth along stream 40.
`FIG. 3D is a block diagram that schematically illustrates
`a file format of a multi-level data stream 41, in accordance
`with another preferred embodimentofthe present invention.
`The data stream is divided into audio slices