(12) Ulllted States Patent
`Fuller et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,711,622 B1
`Mar. 23, 2004
`
`US006711622B1
`
`(54) VIDEO AND AUDIO STREAMING FOR
`MULTIPLE USERS
`
`(75)
`
`Inventors: Michael J_ Fuller, L05 Gatos, CA
`(US); M“ J- Graham» Saratoge CA
`(US)
`
`,
`.
`(73) Asslgnw Bmadware T°°h“°l°g‘°5= Inc->
`C11P6ft1H0> CA(US)
`.
`.
`.
`.
`Subject to any dlsclalmer, the term of thls
`patent is extended or adjusted under 35
`U.S.C. 154(1)) by0 days.
`
`.
`(*) Notlcet
`
`<21) Avv1~N°~= 09/001350
`(22)
`Filed:
`Dec. 31, 1997
`
`5,889,942 A *
`5,905,524 A *
`5,905,872 A
`
`3/1999 Orenshteyn .......... .. 395/187.01
`5/1999 Sauer ........................ .. 348/15
`5/1999 DeSimone et al.
`
`7/1999 Bibayan ................... .. 709/302
`5,922,054 A :
`3:333:33? 2 4 3/333 §i°‘.Z..‘T’.‘.?71..'.‘.
`§5§§§3$
`
`............... .. 709/201
`5,960,173 A *
`9/1999 Tang et al.
`6,006,241 A * 12/1999 Purnaveja et al.
`........ .. 707/512
`6,009,410 A * 12/1999 LeMole et al.
`............. .. 705/14
`6,020,931 A
`2/2000 Bilbrey et al.
`345/327
`6,025,837 A *
`2/2000 Matthews, 111 et al.
`6,055,522 A *
`4/2000 Krishna et al.
`........... .. 705/517
`6,091,407 A
`7/2000 Boetje et al.
`6,151,609 A * 11/2000 Truong ..................... .. 707/505
`6,222,843 B1
`4/2001 Mauger
`21323153331 : 13/588;
`ELV§;11”d‘e‘i‘;“11.‘
`OTHER PUBLICATIONS
`
`.::""""::"333/91??
`
`
`Int. Cl.7 .............................................. .. G06F 15/16
`(51)
`(52) U..S. Cl.
`........................ .. 709/231; 707/505; 379/88
`(58) Fleld Of Search ............................... .. 709/302, 204,
`709/227’ 218’ 231’ 707/104’ 10’ 393/18701’
`343/302’ 329’ 138’ 327’ 703/317’ 348/13’
`379/88
`
`and
`Performance
`Architectures.
`Server
`Video
`SCa1abflityN_Bemhardt’ et al. (1994) flp.eureC0m.fr/ATM/
`paperSEURECOM/Paperg/breStBernhardt.pS.gZ.4
`Realtime Audio and Video Dec. 8 1996‘ http:ei.cs.vt.edu/~
`wwwbtb/book/chap 11/intr’oductio’n.*
`3
`
`(List continued on next page.)
`
`Primary Examiner—Marc D. Thompson
`Assistant Examiner—Thong Vu
`(74) Afmmeyi A8642 0? FiVm—C001€Y G0dWafd LLP
`
`(57)
`
`ABSTRACT
`~
`~
`~
`~
`A. System and method of pmvldmg Streaming and” and
`video ‘data to multiple users is described.
`In one
`eI¥1b0d1m€Ht> the System C0H1Pr1S€S a fir§t Cllent, a S€C9Hd
`client and a server. The first and second clients are executing
`browsers. The server can communicate with the two clients.
`The server concurrently provides streaming audio and video
`data to both of the clients. Importantly, the server does not
`require the two browsers to use a plug-in or a helper
`application to receive and use the streaming audio and video
`data
`'
`
`4 Claims, 5 Drawing Sheets
`
`(56)
`
`References Cited
`
`Us’ PATENT DOCUMENTS
`5,228,859 A *
`7/1993 Rowe ....................... .. 434/118
`5,390,240 A *
`2/1995 Sensney ..
`379/93.14
`5,512,938 A *
`4/1996 Ohno . . . . . . . . .
`. . . . . .. 348/15
`5,602,992 A *
`2/1997 Danneels
`709/248
`5,668,996 A *
`9/1997 Radinsky .................. .. 709/322
`5,724,355 A
`3/1998 Bruno et al.
`.......... .. 707/500.1
`5,751,280 A *
`5/1998 Abbott et al.
`.... N 709/227
`5,764,916 A as
`6/1993 Busey et a1.
`. . . . .. 709/218
`. . . . . .
`5,778,181 A *
`7/1998 Hidary et a1.
`.. 707/104
`.
`5,802,526 A *
`9/1998 Fawcett et al.
`709/204
`5,828,839 A * 10/1998 Moncfeiff -------- --
`-- 709/104
`5:842:05 A : 11/1998 Cunniff 6131-
`--
`5344553 A
`12/1998 H99 et 91‘ “““““““““ " 345/329
`5,861,881 A *
`1/1999 Freeman et al.
`.......... .. 345/302
`5,880,792 A
`3/1999 Ward et al.
`5,884,032 A *
`3/1999 Bateman et al.
`.......... .. 709/204
`5,889,506 A *
`3/1999 Lopresti et al.
`........... .. 345/158
`
`
`
`
`
`/I80
`/- I00
`.
`44 Client side§<— Communications
`Streaming Audio
`and Video Data
`1 76
`I
`.
`/’I76‘StteamingAud%
`Video Data
`5/
`_.I
`\E\\
`:'
`1
`
`
`
`
`
`Browser
`
`112
`178L __
`Streaming Audio and
`Video Data
`'
`
`Browser
`
`III
`
`108
`
`’
`
`Internet
`M
`
`l1
`
`‘
`
`:
`g
`:.
`104
`
`E.
`Video 5
`Display:...
`1
`:4
`11aI
`1
`
`/130
`—>l Server Side >
`I 78 Streaming Audio and Video Data
`I39
`W
`_ .
`0Web Servet
`
`lfl
`
`
`
`
`135
`
`
`
`Real-Time Server HQ
`
`Video
`
`Audio
`
`Video Proxy Audio Proxy
`
`H l P Connection
`to Remote Serve!
`
`
`
`‘—»~ I 75
`I 70
`Line In
`Mike
`Streaming Audio and
`Video Data
`
`L W \166
`
`Samsung-LG-HTC Ex. 1106 p. 1
`
`
`
`Samsung-LG-HTC Ex. 1106 p. 1
`
`

`
`US 6,711,622 B1
`Page 2
`
`OTHER PUBLICATIONS
`
`Instructions for using Netscape Navigator, downloaded from
`Netscape Website/../NetHelp/netscape/navigatr/nav.html,
`printed on Dec. 14, 1997, 67 pages.
`Paek et al. “Video Server Retrieval Scheduling for Variable
`Bit Rate Scalable Video”, IEEE 1996, pp. 108-112.
`
`Huang et al., “The Mercuri Multimedia Laboratory at Hon-
`eywell”, IEEE 1996, pp. 72-77.
`Liu et al Performance of a Storage System for Supporting
`D-ff
`V-d
`T
`d
`1--
`IEEE 1996
`1 erem 1 60
`ypes an Qua mes’
`’ pp.
`1314-1331.
`
`* cited by examiner
`
`Samsung-LG-HTC Ex. 1106 p. 2
`
`Samsung-LG-HTC Ex. 1106 p. 2
`
`

`
`tHCtaP2U
`
`Mar. 23, 2004
`
`Sheet 1 of 5
`
`US 6,711,622 B1
`
`a2\\
`
`%~\\
`
`
`
`AIl|I||l|Iu2mShowTlmaoufiasafioo|VTEmEmmaIIV
`
`
`
`
`
`ESE225bnoum82> QJHotom
`ofifiniom
`
`Q:m:
`
`nouuuanooEm
`
`StowBofium2
`
`n.2w=<59:
`
`R711
`
`on~5“E593%
`
`9:3»
`
`Emu82>
`
`NGE
`
`M5
`
`sun82>
`
`was225mfinaobw
`“.3a:I//1/
`
`39:32>
`
`Q:
`
`EMQou2>waso%:<wfifimobwmmNI\
`
`.3Siam£02?
`
`v.ouou-u.
`
`mi
`
`
`
`.CoEu2vuhfiw
`
`92
`~
`.._________H______.._...____________.____..___¢_—_______.
`
`o€:<mfiaaubm
`5382>Ea
`
`EN
`
`9382>
`
`Ba29%maaafim3~I\_
`
`6885
`
`mm
`
`I
`_____.___._._
`I.._—.
`
`/
`
`uuwsoum
`
`NE.
`
`.wt
`...1/
`
`_._..u..3832>
`..23o§_<magnum
`
`.
`
`.N:
`
`__..___..__.._____.._._____
`
`2:
`
`32>
`
`%m_&m_D
`
`mi:~
`
`“£38m
`
`Samsung-LG-HTC Ex. 1106 p. 3
`
`2:|\
`
`Samsung-LG-HTC Ex. 1106 p. 3
`
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 2 0f5
`
`US 6,711,622 B1
`
`210
`
`Client Initiates HTTP Request
`
`Web Server Supplies Decoding Instructions
`
`.
`
`220
`
`23 0
`
`Client Executes Instructions
`
`240
`
`250
`
`Chent Makes Audio Request
`
`Client Makes Video Request
`
`242
`
`252
`
`Web Server Makes Audio Request
`
`Web Sewer Makes Video Request
`
`2
`
`4
`
`4
`
`2
`
`4
`
`6
`
`254
`
`I
`
`256
`
`Yes
`
`248
`
`Yes
`
`258
`
`Client Dccompresses Audio Data and
`Plays Audio
`
`Plays Video
`
`Client Decomprcsscs Video Data and 2
`
`24]
`
`251
`
`N0
`
`No
`
`FIG. 2
`
`Samsung-LG-HTC Ex. 1106 p. 4
`
`Samsung-LG-HTC Ex. 1106 p. 4
`
`

`
`P.3U.
`
`m3853C.6m3o._.m.mmST\x%
`
`23,2r.aM
`
`
`
` é:=o_s2om%:o_mm_Em:mC.0Em
`
`5cl03teehS
`
`/0SU
`
`6,117,
`
`1
`
`B.-nm.©>~
`
`%%£;~
`
`Q,E.Ew_uu&§..w2.._.fi2whoK:
`
`
`=a.:a.=m=aE»u.95V
`
`I|_!|
`
`D man man DUDDD um}
`
`DEMo=.$_u>5
`
`
`
`V.o__5_u>5
`
`
`
`>.~S332u>_6E2E
`
`
`
`x2_8_.:o=u:<
`
`Euzao9:3.28..
`
`2250£3.329313.0
`
`
`
`Sucau$.50.5__&8_oo.m
`
`EoEmE:o.~Siam
`
`8250mmm3ov=_>>
`
`
`
`Eunau._uuuu.._Em
`
`
`
`353.232o=:owm<
`
`u.=.5mucuubwmom3on=_>»
`
`
`
`Moon30305moEoD82>>52
`
`Samsung-LG-HTC Ex. 1106 p. 5
`
`Samsung-LG-HTC Ex. 1106 p. 5
`
`
`
`

`
`
`
`bmaoum
`
`38:5
`
`U.S.Patent
`..11..._....3..,...:...,1......5:,.......c—._w.um_:Z.>_
`
`
`
`
`.,1;-1$..§.,%.,.
`
`
`
`W¢;.3..;.....rLL..o8L
`
`Mar. 23, 2004
`
`Sheet 4 of 5
`
`US 6,711,622 B1
`
`isEuasm
`
`5E26:am
`
`V2.
`
`32>Eu
`
`
`
`So»umooau
`
`cozuuacou
`
`_u:up._oC.7523N
`
`Joana!5-Cant
`
`
`
`
`
`~22..22E:3.5050.5.5»33¢:=2.=_§
`
`
`
`...:,.m.1...4.=..._....:...rw........:ac.lm.n:..m_..R.u_..N:xm.u..DQM.
`
`
`
`
`
`Samsung-LG-HTC Ex. 1106 p. 6
`
`Samsung-LG-HTC Ex. 1106 p. 6
`
`

`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 5 0f5
`
`US 6,711,622 B1
`
`._omBo._m
`
`32.5.?
`
`82>_S_w_Q
`
`Egan
`
`can82>
`
`._9ou_um
`
`Sw
`
`“amo:z._<
`
`NE.
`
`«S...
`
`3:3uaessmE=8._._
`
`Nr
`
`newE3ea3o5=<
`
`W5‘!
`
`.n._mW¢U
`
`an...
`
`..8o£uwaofiuu3ES
`
`w.U~n~
`
`
`
`
`
`...,..,...w..m...up...........,.uE..w3i£1mm!.€E§$a._..wa2uan,H...n..,.i,u.
`
`
`
`
`
`wrwueu3...atom35¢.WHO«£9»sfiksas.a22.03.
`
`y
`
`C
`
`nuSmaS
`
`6011X.ECTHGL.8
`
`7D...
`
`Samsung-LG-HTC Ex. 1106 p. 7
`
`

`
`US 6,711,622 B1
`
`1
`VIDEO AND AUDIO STREAMING FOR
`MULTIPLE USERS
`
`1. RELATED APPLICATIONS
`
`This application incorporates by reference, and claims
`priority from,
`the U.S. provisional patent application
`entitled, “Streaming Audio and Video for Multiple Users,”
`serial No. 60/034,128, having inventors Michael J. Fuller
`and John J. Graham.
`
`2. THE BACKGROUND OF THE INVENTION
`
`a. The Field of the Invention
`
`This invention relates to the field of network delivery of
`audio and video information. In particular,
`the invention
`relates to a system for delivering audio and video informa-
`tion to multiple users.
`b. Background Information
`The Internet enables many different ways of communi-
`cating. Originally, the Internet was used for the exchange of
`files and electronic mail. As the capabilities of the Internet
`expand, other types of communications are enabled.
`Audio and video transmissions are an important area of
`the communications that the Internet enables. For example,
`many technologies support the transmission of digital video
`and/or audio signals over the Internet. An example of such
`a technology is QuicktimeTM, available from Apple
`Computer, Inc., of Cupertino Calif. Quicktime movies are
`files that can be transmitted across the Internet. Quicktime
`provides both audio and video displays. Many other file
`formats allow audio and video to be displayed on people’s
`computers.
`This paragraph describes an example use of a the Quick-
`time technology. Auser will have a browser application that
`resides on his/her computer. The computer, acting as a client
`under the direction of the browser application, will connect
`to various World Wide Web (web) servers. Each web server
`will typically serve hypertext markup language (HTML)
`files to the clients. The files may include text, graphics
`references, and references to specialized files. Some of these
`specialized files can include audio and video information in
`Quicktime form. The clients can then play these audio and
`video files once they are downloaded using a Quicktime
`plug-in, a helper application, or Quicktime capabilities built
`into the browser application. Aplug-in and a helper appli-
`cation are described in greater detail below.
`Streaming audio and video, as a subset of all the types of
`audio and video that can be transmitted over the Internet,
`allow people to broadcast long and/or live video and audio
`transmissions across the Internet. Streaming video and audio
`is video and audio digital data that is transmitted on a
`continuous basis. A client can access the data stream and
`
`regenerate the video images and audio signal as they are
`being transmitted. Streaming technology is particularly
`helpful where the events are live, or where the files would
`be so large as to be a burden on the end users. Examples of
`where streaming technology is particularly useful are for the
`display of conferences, sporting events, radio broadcasts,
`television broadcasts, and the like.
`RealNetworks, Inc. of Seattle, Wash., provides a system
`for transmitting streaming audio and video signals to users
`over the Internet. RealNetworks supplies a server that allows
`multiple users to simultaneously receive streaming audio
`and video.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`a separate server from their normal web server. For a client
`to receive a real audio broadcast, the client typically con-
`nects through their browser to a Web page with a reference
`to a real audio server. The client then accesses its separate
`real audio player program. The real audio player program
`then connects to the referenced real audio server. A signifi-
`cant drawback to such an arrangement is that the user must
`download the real audio player program. This program must
`then be installed on the user’s computer. This may cause a
`number of problems for the user. For example, if the user is
`behind a firewall, or some security program, the client may
`not be able to receive the broadcast from the server.
`
`the installation of any program may have
`Additionally,
`confiicts with other programs. The program has the disad-
`vantage of being platform specific. This means that a dif-
`ferent program must be developed and downloaded for each
`type of computer that is to be used to access RealNetworks
`broadcasts. Additionally, the broadcasters of the streaming
`audio and video need to use the RealNetworks server, which
`is separate from the broadcasters’ World Wide Web server
`(also referred to as the web server). This increases the
`broadcasters’ security problems because now the broadcast-
`ers must be concerned with two separate servers.
`Another example of a video and audio system that uses
`Internet like communications is the MBone. The MBone is
`
`a specialized communication network that allows for the
`distribution of streaming video and audio signals to multiple
`users. A specialized network is set up specifically to transmit
`MBone communications. A significant drawback of this
`system is that users must be connected to the specialized
`network. Additionally, users will be required to have spe-
`cialized software on their computers to listen to and watch
`MBone transmissions.
`
`A streaming video system, not requiring a user to down-
`load a separate program, was developed for a single user by
`John Graham of California. This single user broadcast
`technology allowed a web server to serve a single streaming
`video signal to a single client. Although the user did not need
`to download a plug-in to see the video, only one user was
`allowed to access the video stream at a time. In this system,
`video information was captured from a video camera and
`digitized. The digital video information was then encapsu-
`lated in a MIME encoded multipart data stream. The client
`received this data stream and reconstructed frames of the
`
`digital video.
`Therefore, what is desired is a platform independent video
`and audio streaming system that does not require the user to
`download additional programs beyond the functionalities
`found in a browser.
`
`3. A SUMMARY OF THE INVENTION
`
`A system and method of providing streaming audio and
`video data to multiple users is described.
`In one
`embodiment, the system comprises a first client, a second
`client and a server. The first and second clients are executing
`browsers. The server can communicate with the two clients.
`
`The server concurrently provides streaming audio and video
`data to both of the clients. Importantly, the server does not
`require the two browsers to use a plug-in or a helper
`application to receive and use the streaming audio and video
`data.
`
`In some embodiments of the invention, a browser causes
`a client to request an HTML file from a web server. The
`client receives the HTML file. The HTML file includes an
`
`The real audio system requires that not only the client
`have additional software, but that the content provider have
`
`HTML tag that directs the browser to load one or more
`applets from the server. The browser executes the applets
`
`Samsung-LG-HTC Ex. 1106 p. 8
`
`Samsung-LG-HTC Ex. 1106 p. 8
`
`

`
`US 6,711,622 B1
`
`3
`causing the browser to request streaming audio and video
`from the web server. That request may or may not include
`parameters giving information about the type of request
`being made. The web server associates a server process with
`the request, given the parameters in the request. The web
`server notifies real-time audio and video process that stream-
`ing audio and video information is needed. In response to the
`notification, the real-time audio and video process stores
`encoded audio and video data in a shared memory location.
`The server process accesses the shared memory and inserts
`the audio and video data into one or more data streams. The
`client receives the data streams and reconstructs the audio
`
`5
`
`10
`
`and video signals using only the capabilities of the browser.
`In some embodiments, a separate stream and server process
`is used for each of the audio and video data. These embodi-
`
`15
`
`ments allow multiple clients to simultaneously receive the
`same audio and video data.
`Other embodiments of the invention include a web server
`
`that can serve streaming audio and video information as well
`as perform more usual web server functions (such as,
`serving web pages, performing file transfers, supporting
`secure communications). These embodiments have the
`advantage of allowing the broadcasters and the users to set
`up their security configurations for one web server, rather
`than two servers (a web server and a streaming audio and
`video server).
`Although many details have been included in the descrip-
`tion and the figures, the invention is defined by the scope of
`the claims. Only limitations found in those claims apply to
`the invention.
`
`4. A BRIEF DESCRIPTION OF THE DRAWINGS
`
`The figures illustrate the invention by way of example,
`and not limitation. Like references indicate similar elements.
`
`FIG. 1 illustrates a system including one embodiment of
`a streaming audio and video system for multiple users where
`client computers do not need plug-ins or helper programs.
`FIG. 2 illustrates an example method of streaming audio
`and video for multiple users.
`FIG. 3 illustrates a web page that a user can use to access
`a streaming audio and video broadcast.
`FIG. 4 illustrates a web page for selecting the bandwidth
`of the user’s Internet connection.
`
`FIG. 5 illustrates a web page having streaming audio and
`video.
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`5. THE DESCRIPTION
`
`4
`Web server: a server for serving at least Internet related
`requests. Example web servers can serve HTML pages
`in response to HTTP requests from clients. Some web
`servers can serve many different kinds of requests, e.g.,
`HTTPS, and/or FTP.
`Browser: an application, program, process, or combina-
`tion of applications, programs or processes, that allow
`a client to make a request of a web server and process
`the results of the request. The browser may be part of
`a stand alone application or a set of programs that are
`integrated into the operating system of the client.
`Plug-in: plug-ins are external software programs that
`extend the capabilities of the browser in a specific way.
`For example, a plug-in can be used to give the browser
`the ability to play audio samples or view video movies
`from within the browser.
`
`Helper application: like plug-ins, helper applications are
`external software programs. The browser redirects
`some types of file to the helper applications. The helper
`applications allow clients to process many different
`types of files on the Internet. When the browser
`encounters a sound, image, or video file, the browser
`hands off the data to the helper applications to run or
`display the file.
`JavaScript: a stand-alone programming language built
`into many browsers. Primarily an extension to the
`Internet standard HTML language.
`Java: a programming language whose programs (called
`applets) can travel over the Internet for use by clients.
`Whereas plug-ins and helper applications are tailored
`for a particular type of client, Java applets generally
`work across the Internet regardless of the type of client.
`Many browsers include Java capabilities so Java
`applets require no installation. Except where noted,
`Java and JavaScript are interchangeable.
`b. System
`FIG. 1 illustrates a system including one embodiment of
`the invention where audio and video is supplied over the
`Internet to multiple users. The following paragraphs first list
`the elements of FIG. 1, describe their interconnections, and
`then describe the various elements in greater detail.
`This paragraph lists the elements of FIG. 1. FIG. 1
`includes a system having three parts: a client side 100, a
`communications interface 180, and a server side 130. The
`client side 100 includes a client 112 and a client 111. The
`
`client 112 includes a browser 102 having a video display
`area 103. The client 111 includes a browser 108 and a video
`
`50
`
`display area 104. The communications interface 180
`includes the Internet 185. The server side 130 includes a web
`
`server 131, a shared memory 135, and a real-time server
`140. The web server 131 includes two processes, a process
`138 and a process 139. The real-time server 140 includes a
`video module 144, an audio module 146, a video proxy 148,
`and an audio proxy 149. The real-time server 140 interfaces
`with a number of other elements. These elements include a
`
`video card 159, an input audio interface 162, and an HTTP
`connection to remote server 170. Various elements of FIG.
`1 have now been listed.
`
`The following paragraphs describe the interconnections
`between the elements of FIG. 1. Beginning on the server side
`130, the video card 159 receives a video input 158 outputs
`a digital video signal to the video module 144. Similarly, an
`input audio interface 162 receives a mike input 164 and/or
`a line input 166 and outputs a digital audio signal to the
`audio module 146. The HTTP connection to remote server
`
`The following sections describe embodiments of the
`invention. The first section provides definitions that will help
`in the understanding of the remaining sections. The second
`section shows an example system that supports various
`embodiments of the invention. The third section describes an
`
`example method of using the invention. The fourth section
`illustrates an actual use of the streaming audio and video
`used in some embodiments of the invention. The last section
`reviews additional alternative embodiments of the invention.
`a. Definitions
`
`The following definitions will help in the understanding
`of the following description.
`Client: a computer, process, program, or combination of
`computers, processes or programs, that can communi-
`cate with a server.
`
`55
`
`60
`
`Server: a computer, process, program, or combination of
`computers, processes or programs, that can communi-
`cate with a client. The server and the client can be
`
`65
`
`executing on the same computer(s).
`
`170 receives a streaming audio and video data 175. The
`
`Samsung-LG-HTC Ex. 1106 p. 9
`
`Samsung-LG-HTC Ex. 1106 p. 9
`
`

`
`US 6,711,622 B1
`
`5
`HTTP connection to remote server 170 outputs the video
`data to the video proxy module 148 and the audio data to the
`audio proxy module 149. The real-time server 140 uses the
`data received by the various modules and stores portions of
`that data in the shared memory 135, after some manipulation
`of the data. The shared memory 135 is accessed by the web
`server 131. The process 138 and the process 139 transmit
`and receive data to the communications interface 180. Thus,
`the couplings of the server side 130 have been described. In
`some embodiments, the process 138 and the process 139
`correspond to HTTPD processes.
`The communications interface 180 allows the client side
`100 to communicate with the server side 130. The commu-
`
`nications interface, and the Internet 185 in particular, sup-
`port many different types of connections by the client side
`100 and the server side 130. In the example communications
`inter face 180, the communications interface 180 includes
`the Internet 185. In particular, in this example, the process
`138 is communicating streaming audio and video data 176
`to the Internet 185. Similarly, the process 139 is communi-
`cating the streaming audio and video data 178 to the Internet
`185.
`
`On the client side 100, the client 112 is communicating
`the streaming audio and video data 176 with the Internet
`185. Similarly,
`the client 111 is communicating with the
`Internet 185 to receive and manage the streaming audio and
`video data 178. The clients then communicate the informa-
`
`tion to their respective browser applications. The browser
`applications the generate video images in their respective
`video display areas.
`Thus, the connections between the various elements of
`FIG. 1 have been described. Now the various elements will
`
`be described in greater detail in the following paragraphs.
`An example method of using these elements is described
`below in relation to FIG. 2.
`The server side 130 will be described first.
`
`The video input 158 represents a video signal that a user
`of such a system wishes to broadcast to the clients on the
`client side 100. The video input 158 can include analog
`signals representing video information. The video card 159
`digitizes the video input 158 to produce a digital video
`image. Various embodiments of the invention include Sun
`video cards available from Sun Microsystems, of Mountain
`View, Calif., and Parallax XVideo XtraTM video cards,
`available from Parallax Graphics, Inc., of Santa Clara, Calif.
`However, what is important is that the real-time server 140,
`and in particular the video module 144, receives some sort
`of digitized video signal. The video module 144 is respon-
`sible for providing the real-time server 140 with the video
`information that will be broadcast to the client side 100. The
`
`video module 144 can convert the digitized video signals to
`be of better use to the rest of the system. For example, the
`video module 144 may, if not done by the video card 159,
`convert digital video data into a sequence of JPEG digital
`video frames. In any case, what is important is that the
`real-time server 140 receives digital video information in a
`format
`that it can use (example formats include, JPEG,
`MPEG, GIF, and AVI).
`Similarly,
`the input audio interface 162 allows for the
`input of analog audio signals and converts this input to
`digital audio signals. What is important is that the audio
`module 146 receives a digitized audio signal that can be used
`by the real-time server 140. The audio module 146 may
`convert the digitized audio signal into any of a number of
`formats, corresponding to any of a number of audio trans-
`mission rates (examples include G.711 and G.723 audio
`compression formats).
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`The HTTP connection to remote server 170 represents an
`important advantage of one embodiment of the invention. In
`this embodiment, the HTTP connection to remote server 170
`allows the server side 130 to forward broadcasts of audio
`
`and video signals from other streaming audio and video
`servers. In these uses, the server side 130 acts as a client to
`another server. The HTTP connection to remote server 170
`
`can receive video and audio signals being broadcast through
`the Internet 185 from another server. The HTTP connection
`
`to remote server 170 provides the digital video information
`from the other server to the video proxy module 148.
`Similarly, the HTTP connection to remote server 170 pro-
`vides the audio data to the audio proxy module 149. The
`video proxy module 148 and the audio proxy module 149
`then supply the respective video and audio data to the
`real-time server 140.
`
`The real-time server 140 represents an application, or set
`of applications, executing on one or more computers, that
`prepares audio and video data for broadcasting to multiple
`users through the web server 131. The real-time server 140
`takes the data from the various modules, processes the data,
`and stores the processed data in the shared memory 135. The
`real-time server 140 can perform compression, and other
`manipulations of the data, to reduce the processing burden
`on the web server 131. For example, in some embodiments
`of the invention, the real-time server 140 receives digitized
`video data and compresses that data into JPEG images.
`These JPEG images are sequenced digital frames of video.
`Similarly, for the audio data, the real-time server 140 breaks
`the audio information into one-half second time periods of
`audio data (other embodiments use other time periods).
`These one-half second time periods of data are stored in the
`shared memory 135. The real-time server 140 can also
`compress the audio information into one of a number of
`various compressed audio signals (e.g., G.711 and/or G.723
`audio compression formats). In some embodiments of the
`invention, the real-time server can broadcast audio and video
`from multiple sources to multiple clients.
`The shared memory 135 represents a shared storage area
`for use by the real-time server to store audio and video data
`for access by the web server 131. In one embodiment the
`shared memory 135 has a locking and semaphore usage
`scheme to ensure that the real-time server 140 is not writing
`data into the shared memory 135 while the web server 131
`is accessing that data. In some embodiments,
`the sema-
`phores act as notifiers to indicate that new data in the shared
`memory 135 is available for use by the web server 131. In
`some embodiments, the video data and the audio data are
`stored in different shared memory locations.
`The web server 131 communicates data over the Internet
`
`185 using one or more communications protocols. In some
`embodiments of the invention,
`these protocols include
`HTTP (Hypertext Transfer Protocol), TCP (Transmission
`Control Protocol) and UDP (User Datagram Protocol). The
`web server 131 represents an application, including one or
`more processes, for communicating over the Internet 185. In
`one embodiment, the web server 131 includes an Apache
`web server. Each of the processes in the web server 131
`represents one or more processes for serving streaming
`audio and video data to the client side 100.
`In some
`embodiments, the web server 131 transmits the video data as
`a multipart MIME (multi-purpose Internet mail extensions)
`encoded file for decoding directly by the browser or as
`compressed video information for decoding by an applet run
`in the browser. The web server 131 transmits the audio data
`
`as compressed audio data for decoding by an applet run in
`the browser.
`
`Samsung-LG-HTC Ex. 1106 p. 10
`
`Samsung-LG-HTC Ex. 1106 p. 10
`
`

`
`US 6,711,622 B1
`
`7
`the web server 131 initiates a
`In some embodiments,
`separate process for each audio and video connection made
`from client side 100. Thus, for one client receiving stream-
`ing audio and video data, two processes would be started
`within the web server 131. The first process would supply
`video data and the second process would supply audio data.
`The processes access the shared memory 135 and serve the
`data across the Internet to the respective client.
`In some embodiments, the web server 131 initiates at least
`one process for each client. This provides important advan-
`tages in some embodiments of the invention. In particular,
`because the web server 131 is serving the data directly
`through processes it created, server side 130 users need not
`worry about security issues beyond those already faced with
`their web server 131. Thus,
`these embodiments of the
`invention have a lower chance of interfering with client side
`100 fire walls and have a lower chance of having a server
`side 130 security problem.
`Other embodiments of the invention include separate
`Common Gateway Interface (CGI) programs for audio and
`video. These CGI programs are used by the web server 131
`to serve the streaming audio and video data. These CGI
`programs are not necessarily integrated as tightly to the web
`server 131 as the process 138 and the process 139. However,
`a CGI program allows for the easy extension of many
`different types of web servers.
`The communications interface 180 permits communica-
`tions between the server side 130 and the client side 100. In
`
`this example, the Internet 185 supports the communications.
`Other embodiments of the invention support other commu-
`nications interfaces. For example, the Internet 185 can be
`replaced by a local area network, a wide are network, a
`proprietary telecommunications and networking
`infrastructure, or some other communications interface.
`What is important is that the server side 130 can commu-
`nicate with the client side 100. The communications inter-
`face 180 can also include combinations of the above
`
`described technologies. For example, the Internet 185 can
`include a web server to which the clients on the client side
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`communicate through to access the Internet 185. The client
`side 100 can be on a local area network that is connected
`
`40
`
`through a server, or router, to the Internet 185.
`The client side 100 represents the consumers of the
`streaming audio and video data. In this example, the two
`clients are receiving separated streaming audio and video
`data signals. Other embodiments of the invention support
`many more clients.
`The client 112 represents a computer, such as a PC
`compatible computer, running a browser application 102.
`For video display, the browser application 102 can include
`a Netscape Navigator” or CommunicatorTM program for
`“multipart/x-mixed-replace MIME type video,” or a
`Microsoft Internet Explorer” 3.0 or later for a Java based
`video transmission. In some embodiments, the Java based
`video transmission applet parses the multipart/x-mixed-
`replace MIME type video. For audio, the browser applica-
`tion 102 can include any browser that supports Java and/or
`JavaScript.
`The browser application 102 is responsible for receiving
`the streaming audio and video data 176 and reconstructing
`an audio and video signal suitable for the end user. In this
`example,
`the video display area 103 displays the recon-
`structed video information received from the video input
`158 at the real-time server 140. Similarly, the client 111 is
`executing the browser 108. The browser 108 is displaying
`the same video signal in the video display area 104. The
`client 11 represents another computer executing a browser
`application.
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`Various embodiments of the invention have modifications
`to the system shown in FIG. 1. Some of these v