(12) UnIted States Patent
`(10) Patent N0.:
`US 8,156,236 B2
`
`Costanzo et a].
`(45) Date of Patent:
`*Apr. 10, 2012
`
`U3008156236B2
`
`(54) AUDIO-VIDEO DATA SWITCHINGAND
`.
`,
`VIEWING SYSTEM
`IIIVCHIOISI Filippo Costanzo, L05 Angel“, CA
`(US); Saverio Roncolini, Rome (IT);
`Antonio Rossi, Rome (IT)
`
`(76)
`
`(,1, )
`
`NOIiCCZ
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C.1'4
`b 86 da '.
`3 (b) y
`yb
`This patent is subject to a terminal dis-
`clalmcr’
`
`EP
`
`(21) APPL N05 12’497’273
`
`(22) Filed:
`
`Jul. 2, 2009
`
`(65)
`
`Prior Publication Data
`US 2010/0146576Al
`Jun. 10, 2010
`
`Related U.S.Appllcat10n Data
`(63) Continuation of application No. 09/897,708, filed on
`Jul. 2, 2001, now Pat. No. 7,571,244.
`$203188%na1 application No. 60/218’507’ filed on JUL
`7’
`‘
`
`(60)
`
`
`
`5,238,332 :
`5,
`,
`5,600,368 A
`5,613,122 A
`5,633,810 A
`5,649,105 A
`5,706,054 A
`5,719,786 A ,
`5,745,126 A '
`
`123/332 11301116618}
`7335/53):
`/_
`ey eta.
`..
`
`2/1997 Matthews, HI
`.. 348/143
`3/1997 Burnard et al.
`..
`713/1
`
`.. 370/431
`5/1997 Mandal et a].
`
`7/1997 Aldred et a1.
`.. 709/220
`
`75/24012
`1/1998 Hannah .....
`2/1998 Nelson et al.
`..... 709/219
`
`4/1998 dam et a1.
`382/154
`
`(ContInued)
`
`
`FOREIGN PATle DOCUMJNTS
`0847200 A1
`6/1998
`(Continued)
`
`OTHER PUBLICATIONS
`
`Kunta, Mike “Re: Question on 18" Dish Service Providers!” Google-
`Groups Newsgroup:althomcrcpair. Dec. 27, 1997.
`on 1nuc
`(C t'
`d)
`
`Primary Examiner 7 Ajay Bhatia
`(74) Attorney, Agent, or Firm iNorman R, Van 'l'reeck;
`Sheldon Mak & Anderson
`(57)
`ABSTRACT
`A computer system and method providing for viewing and
`switching ofaudio-video data. The system comprises: a plu-
`
`rality Ofaudio/video sources containinginformation referring
`to an event; a streamIng server, streamIng the contents of a
`first audio signal and a first video signal from the audio and
`video sources to auser; a feed distributor controllably feeding
`the first audio signal and first video signal to the streaming
`server; and a user-operated control unit communicating with
`t e
`ee
`Istr1 utor an
`contro Inc 0 eration o t e ee
`h f d d.
`.b
`d
`11. a
`p
`i
`f h f d
`distributor, so as to instruct the feed distributor to switch
`between audio or Video. Switching between audio signals
`occurs without altering the video signals and switching
`between video signals occurs without altering the audio sig—
`nals.
`
`46 Claims, 3 Drawing Sheets
`
`(2006.01)
`
`(51)
`
`(56)
`
`Int. Cl.
`G06F 15/16
`709/231
`(52) U S Cl
`58) F: Iii
`f Clfit"""S Mini];""""""""""" 709 ,231
`(
`'
`gm 01.
`I
`:5“ 1;: [fun
`earc] t """""11 t
`ee app 1ca 1011
`e or comp e e searc 1 ns ory.
`,
`.
`References Clted
`U.S. PATENT DOCUMENTS
`4,037,250 A
`7/1977 McGahan et a1,
`4,814,869 A
`3/1989 Oliver, Jr,
`5,170,252 A "‘
`12/1992 Gearetal.
`5,410,698 A
`4/1995 Danneels et al.
`5,539,483 A
`7/1996 Nalwa
`
`..
`
`
`
`348/159
`348/159
`348/659
`719/331
`
`
`
` SERVER
`SESSION
`MANAGER
`
`
`
`THEATRE
`'
`INTERFACE
`
`
`BUILDER
`DESCRIPTOR
`
`
`VtDEO
`
`WINDOW
`
`
`36
`
`STREAM
`
`STREAM
`
`i RENDEREH
`37
`
`._.
`CONSUMER '
`
`CST000834
`
`NEULION 1001
`
`CST000834
`
`1
`
`NEULION 1001
`
`

`

`US 8,156,236 B2
`
`PageZ
`
`U.S. PATENT DOCUMENTS
`
`5,761,416 A
`_
`5,7/4,664 A
`5,805,763 A
`A
`5,841,432 A
`5,850,352 A
`..
`5,854,893 A +
`5,854,898 A *
`,_ ,,
`5,872,887 A *
`5,894,320 A
`5,930,473 A
`5,933,603 A
`5,949,792 A
`59965015 A
`6,008,867 A
`6,014,706 A
`6,035,341 A
`6,061,732 A
`6,122,668 A
`6,154,771 A *
`6,195,680 B1
`6,208,335 B1
`6,233,428 B1
`6,236,395 B1
`6,269,394 B1
`6,310,661 B1
`6,326,982 B1
`6,356,294 B1
`6,418,473 B1
`
`
`
`
`6/1998 Mandaletal,
`,,,,,,,,,,,,,,,, 709/250
`,
`.
`61998 H1da1yetal.
`..
`725/110
`,
`9/1998 Lawleretal.
`..
`386/83
`
`,
`11/1998 Carmeletal,
`715/5001
`.
`,
`.
`
`12/1998 Moezzr et a1,
`,. 345/419
`.
`,
`.
`12/1998 Ludw1g et a1
`,. 709/231
`,
`.
`12/1998 R1dd1e
`,. 709/231
`,
`,
`,
`,
`219% Walker
`,. 386/341
`,
`41999 Vancelette .
`,. 725/138
`,
`71999 Teng etal.
`,
`., 709/204
`,
`.
`8/1999 Vahaha etal
`,. 709/225
`,
`91999 Yasuda et a1.
`,. 370/474
`,
`”/1999 Day 6“”
`" 709/226
`12/1999 Cooper etal,
`,. 348/705
`
`1/2000 Cannon etal.
`.
`,. 709/231
`
`3/2000 Nunally et a1.
`,. 709/253
`,. 709/231
`5/2000 Korst ct a1.
`
`,
`., 709/231
`9/2000 Teng etal.
`11/2000 Rangan etal.
`,. 709/217
`2/2001 Goldszmidt et a1,
`,. 709/203
`3/2001 Gordon et 31.
`.
`,. 345/721
`
`5/2001 Fryer
`,. 434/308
`5/2001 Sezan et 31.
`345/723
`709/217
`7/2001 Kenner et a1.
`
`10/2001 Arscnault
`..
`,. 348/725
`
`12/2001 Wu et 31.
`,. 345/718
`348/1407
`,,
`3/2002 Martin et a1,
`
`7/2002 St. Malu'ice Ct :11,
`,. 709/231
`
`6,510,553 B1
`
`,
`,
`
`,
`
`JP
`W0
`W0
`
`
`
`.
`.
`
`
`
`1/2003 Hazra ......................... . 725/87
`
`6578201 B1
`@003 ”Rimmed
`‘ 72/5/86
`6,681,398 B1
`1/2004 Verna ........
`725/141
`/
`6,711,622 B1
`3/2004 Fuller et al.
`709/231
`,
`6,732,183 B1
`5/2004 Graham
`709/231
`,
`6,744,463 B2
`..
`6/2004 Rye etal.
`348/159
`
`,
`6,757,305 B1
`..
`6/2004 Soepenberg et a1.
`370/536
`,
`6,763,377 B1
`7/2004 Belknap et a1.
`............... 709/231
`.
`7,047,305 131
`5/2006 Brooksetal,
`
`5
`,
`2001/0047423 A1
`11/2001 Shae et a1.
`709/235
`,1
`,
`/
`2002/0007493 A1
`725/109
`1/2002 Butleret a1.
`5
`/
`2002/0038374 A1
`3/2002 Gupta et a1.
`709/231
`,;,
`-
`,
`2002/0107940 A1
`709/231
`8/2002 Brassr1
`,.
`,
`2004/0143675 A1
`709231
`7/2004 Aust
`,,,,,,
`2008/0285948 AH 11/2008 Iwase et a1.
`..
`...... 386/95
`
`
`
`FOREIGN PATJNl DOCUM,:NTS
`,
`51,1998
`410536277 A
`331999
`9912349
`@001
`0172014
`,,
`,
`OTHER PUBLILATIONS
`Warnock, Todd, “PrimeStar4000d experiences and a good deal”
`Google-Groups Newsgroup: rec.Video,sate111te,dbs, Dec. 7, 1994.
`Martzke, Rudy: Instant replay gets “Matrix-like View of key plays",
`USATODAYCDJII Jan, 24, 2001
`
`* cited by examiner
`
`CST000835
`
`CST000835
`
`2
`
`

`

`US. Patent
`
`A
`
`m2w,
`
`00S
`
`2B6
`
`U_
`
`_
`
`3im,#2«6.5w,_..0.“—
`
`Wlllllllll
`
`
`
`2szfiEm_—l
`
`«M552%.Ill
`
`1F_smab_‘w
`_.........____3.52m;8_..ox----
`.M--
`
`
`
`mm>mmmPZMZO
`
`CST000836
`
`CST000836
`
`3
`
`
`

`

`US. Patent
`
`Apr. 10, 2012
`
`Sheet 2 of 3
`
`US 8,156,236 B2
`
`9.
`
`5
`
`mugs—firth
`
`mw>mmm
`
`mp
`
`50
`
`mm0<z<§
`
`GZ_S_<wm._.w
`
`#2m.30
`
`t.
`
`Om9>
`
`>>ODZS>
`
`om
`
`CST000837
`
`CST000837
`
`4
`
`
`
`
`
`

`

`US. Patent
`
`2B632,
`
`S_3J_MZ2_H.9onaw.1“:_h.uIIIIIII!IIII.IIll-Sp_
`
`
`
`
`P4._.A.III|I[III-[Il-l-lI-lolllluLmFPI-
`s'll._A:_EI.E.EE__«F«FESHm_<,_.llil.llilil.
`
`
`
` 2.I:-_-Egan;-M,_mmofimfiwwmmwm‘mmsmmm530522%.n._$2memm>mmwqFzm>mEm:
`
`
`
`
`
`Emmfi_.mmznmzoommmwozmm._Emma
`
`6Iwm0....—
`
`
`
`8,.8BS1.1!:1.211!...|.
`
`CST000838
`
`CST000838
`
`5
`
`
`

`

`US 8,156,236 B2
`
`1
`AUDIO-VIDEO DATA SWITCHING AND
`VIEWING SYSTEM
`
`
`
`CROSS-REFERENCE
`
`This application is a continuation of application Ser. No.
`09/897,708, filed Jul. 2, 2001, now US. Pat. No. 7,571,244,
`which is incorporated herein by reference, which claims the
`benefit of Provisional Application No. 60/218,507 filed on
`Jul. 15, 2000.
`
`10
`
`FIELD OF THE INVENTION
`
`invention relates to webcast streaming of
`The present
`audio—visual events. More specifically, the invention relates to
`an audio—video data switching and viewing system which
`allows viewing and smooth remote switching from one video
`signal to another or from one audio signal to another.
`
`BACKGROUND OF TI IE INVENTION
`
`Fri or Art
`
`2
`cameras (different points ofview) adopted. Therefore, a high
`bandwidth is required in order to obtain an audio—visual con—
`tent of a good quality.
`In order to solve such problem, a different session for each
`View could be established. This means that only a single
`audio—visual content at the time would be streamed and, each
`time a client desires to switch from one view to another, the
`streaming server 1 would pick a different file and retransmit it
`to the client. Such teclmology is, for example, adopted in the
`“BigBrother” series, when transmitted over the Internet. See,
`for example, http://www.endemol.com or http://www.cb-
`s.com/primetime/bigbrother. While this solution allows a
`larger bandwidth, the switching delay is unacceptable for the
`user. In fact, according to the usual way of streaming signals,
`a first step ofthe streaming process is that ofbuffering data on
`the client computer. Then, after a predetermined amount of
`time, the data are shown 011 the screen of the client while, at
`the same time, the remaining data are being transferred over
`the c01mection. This means that, each time a switching
`occurs, a considerable amount of time would be spent in
`buffering again the audio-visual data of the following stream,
`with a delay which would be unacceptable for most kind of
`commercial applications and which would result in an inter-
`ruption of both the audio and the visual content of the signal
`transmitted on the screen.
`
`SUMMARY OF THE INVENTION
`
`The present invention solves the prior art problems cited
`above, by allowing each user to remote controlling between
`different cameras, thus creating a customized show with a
`seamless switching and optimal use of bandwidth. More spe-
`cifically, when switching among diiferent points of View, the
`system according to the present invention is such that neither
`audio nor video interruptions occur, and the new view
`replaces the old one with a perfect transition,
`According to a first aspect, the present invention provides
`a computer system for viewing and switching of audio-video
`data, comprising: a plurality of audio and video sources con-
`taining information referring to an event; a streaming server,
`streaming the contents of a first audio signal and a first video
`signal from the audio and video sources to a user; a feed
`distributor, connected between the audio and video sources
`and the streaming server, the feed distributor controllably
`feeding the first audio signal and first video signal to the
`streaming server; and a user-operated control unit communi-
`cating with the feed distributor and controlling operation of
`the feed distributor, so as to instruct the feed distributor to
`switch between video signals whereby, upon switching, the
`feed distributor feeds to the streaming server a second video
`signal which is different from the first video signal without
`altering the first audio signal.
`According to a second aspect, the present invention pro-
`vides a computer system for viewing and switching ofaudio—
`video data, comprising: a plurality ofaudio and video sources
`containing information referring to an event; a streaming
`server, streaming the contents of a first audio signal and a first
`video signal from the audio and video sources to a user; a feed
`distributor, connected between the audio and video sources
`and the streaming server, the feed distributor controllably
`feeding the first audio signal and first video signal to the
`streaming server; and a user-operated control unit communi-
`cating with the feed distributor and controlling operation of
`the feed distributor, so as to instruct the feed distributor to
`switch between audio signals whereby, upon switching, the
`
`CST000839
`
`30
`
`35
`
`4o
`
`45
`
`According to the webcast streaming technology, a client-
`scrver c01mection is established, where the server transmits
`multiple streams or files to each client. Each stream or file
`relates to a different point of view. Each stream or file is
`output either from stored files or from live encoded feeds, for
`example by means of encoding stations.
`FIG. 1 shows an exemplary embodiment of such prior art
`system. Products embodying such technology are,
`for
`example, produced by the company iMove Inc., and shown at
`the website address http://www.imoveinc.com. A streaming
`server 1 located on the server side receives audio—visual infor—
`mation from a number of different audio—visual files or
`streams connected to the source of information, such as an
`audio file FA and video files FVl .
`.
`. FVn. all indicated with
`2 in the Figure.
`The audio-visual content of the numbern offiles 2 (three in
`the example) is streamed from the server to the client over a
`connection 3. The connection 3 is an Internet connection. As
`a consequence, it can assemble different network technolo-
`gies, such as Ethernet, Frame Relay, ATM switch, CDN,
`satellite uplink and downlink, DSl, D2, D53 (or the corre-
`sponding European E1, E2, E3), fiber, modem, ISDN, xDSL,
`and so on. All these technologies use the IP protocol and are
`interconnected by routers, bridges and gateways. Assuming
`that the maximum available bandwidth for the connection is ,
`b, the maximum bandwidth for each streamed file will be b/3.
`On the client side, a streaming client software 4 provides
`for the interpretation of the received streams. One of the
`streams is shown on the screen ofthe client in a current view,
`For example, the contents relating to the video file FVZ can be
`shown, as indicated by the box 5, represented in solid lines
`and relating to the “current view (2)”, namely the view relat-
`ing to the contents of FVZ.
`As soon as the viewer wants to switch on a different point
`of View, he will send a command to the GUI (graphic user
`interface) 6. for example by means of a pointing device (not
`shown in the Figure), and from the GUI 6 to the streaming
`client 4. As a result, the audio-visual content shown on the
`screen will from now on relate for example to the contents of
`FV1, indicated by the box 7, represented in dotted lines.
`A problem of the prior art shown in FIG. 1 is that the
`required bandwidth is directly proportional to the number of
`
`60
`
`65
`
`CST000839
`
`6
`
`

`

`US 8,156,236 B2
`
`5
`
`10
`
`15
`
`,
`
`30
`
`35
`
`4o
`
`45
`
`3
`feed distributor feeds to the streaming server a second audio
`signal which is different from the first audio signal without
`altering the first video signal.
`According to a third aspect. the present invention provides
`a computer-operated method for viewing and switching of
`audio—video data, comprising the steps of: providing a plural—
`ity of audio and video sources containing information refer-
`ring to an event; streaming contents of a first audio signal and
`a first video signal from the audio and video sources to a user;
`controlling the streaming of Video signals, so as to switch
`between video signals, streaming, upon switching, a second
`video signal which is different from the first video signal
`without altering the first audio signal.
`According to a fourth aspect, the present invention pro-
`vides a computer-operated method for viewing and switching
`of audio—video data, comprising the steps of: providing a
`plurality of audio and video sources containing information
`referring to an event; streaming contents ofa first audio signal
`and a first video signal from the audio and video sources to a
`user; controlling the streaming of audio signals, so as to
`switch between audio signals, streaming, upon switching, a
`second audio signal which is different from the first audio
`signal without altering the first video signal.
`Advantageous embodiments of the present invention are
`claimed in the attached dependent claims.
`The present invention overcomes the problems ofthe prior
`art in several aspects: first, the bandwidth is not wasted as
`done with prior art systems. The Internet connection carries,
`at every time, only one video stream and one audio stream. As
`a consequence, a virtually unlimited number of different
`points of view can be used. Second, the audio signal is not
`interrupted during switching, Third, there is a smooth Video
`transition on the screen ofthe user between different points of
`View.
`In accordance with the present invention, there is no need
`to establish a new session over a new connection each time a
`switching of point of View occurs.
`The present invention is particularly advantageous in a
`system requiring a high number of cameras, like for example
`from 30 to 50 cameras. Such high number of cameras shoot-
`ing an event, provides the user with a sort ofa virtually infinite
`camera, the cameras being arranged with the correct parallax
`in a matrix fashion. In this case, a system like the system
`described in FIG. 1 cannot be implemented. By contrast, this
`case is well suited to the system according to the present
`invention, where the occupied bandwidth is independent
`from the number of different cameras.
`Other features and advantages ofthe invention will become
`apparent to one skilled in the art upon examination of the
`following drawings and detailed description. It is intended
`that all such adcitional features and advantages be included
`herein within the scope of the invention, as is defined by the ,
`claims.
`
`
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention Will be understood in better detail
`with reference to the attached drawings, where:
`FIG. 1 shows a prior art system, already described above;
`FIG. 2 is a schematic diagram of the system according to
`the present invention; and
`FIG. 3 describes in greater de ail the diagram shown in
`FIG. 2.
`
`
`DETAII
`I) D ‘SCRIPIION ()F III; PREFERRED
`
`EMBODIMENT
`
`
`
`6O
`
`65
`
`FIG. 2 shows a schematic diagram of the system according
`to the present invention. According to the present invention,
`
`4
`the streaming server 11 on the server side is not directly
`connected to the audio—visual sources 12. In particular, a feed
`distributor 13 is present, connected between the audio—visual
`files 12 and the streaming server 11. The feed distributor 13
`receives instructions from the GUI manager 14 located on the
`client side. The GUI manager 14 receives inputs from an
`active GUI 15, also located on the client side. The GUI man—
`ager 14 on the client side is distinct from the streaming client
`software 17 for processing the audio-video data streamed
`from the server. The streamed contents are shown on the
`client screen inside a video window 50. The GUI manager 14
`is a user-operated control unit. The instructions from the GUI
`manager 14 to the feed distributor 13 are transmitted along a
`connection 16.A client proxy 21 and a server stub 30 are also
`shown, located between the GUI manager 14 and the feed
`distributor 13, and will be later described in better detail.
`As also explained later, the feed distributor 13 could be
`implemented either on a computer which is separate from the
`computer containing the streaming server, or on the computer
`containing the streaming server. In the preferred embodiment
`of the present application, the streaming server and the feed
`distributor are on the same computer.
`A first embodiment ofthe present invention provides trans—
`mitting only a single stream of audio-visual data (coming for
`example from the video file FV1 and also comprising the
`audio file FA) along a connection 18 between the streaming
`server 11 and the streaming client 17. A second embodiment
`could provide a main stream of audio—visual data output on a
`main Window ofthe user, and a plurality of accessory streams
`output on secondary windows (thumbnails), wherein the
`accessory streams have an extremely reduced bandwidth
`occupation and wherein the audio—visual contents ofthe main
`window can be switched by the user according to the present
`invention.
`During operation, as soon as the user wishes to change
`from a first point of View to a second point ofView, switching
`for example from the video file FVl to the video file FV2, the
`active GUI 15 instructs the GUI manager 14, which in turn
`instructs the feed distributor 13 on the server side to switch
`between video files. Upon receipt of such instructions, the
`feed distributor 13 selects the video file VF2 and transmits
`this file to the streaming server 11. During the switching of
`points of View, the audio fileiwhich is usually interleaved
`with the video file during the streaming operationiis not
`altered. Thus, no audio switching occurs when changing view
`from one camera to another. Moreover, according to a pre—
`ferred embodiment of the present invention, the video switch-
`ing between points of view occurs in a smooth maimer. Dif-
`ferently from what disclosed in the prior art ofFIG. 1, here, a
`switching command by the user causes a switch on the server
`side, so that the streaming server 11 streams a signal which is
`different from the signal which was streamed before the
`switching command. Further, differently from what disclosed
`in the prior art like the Internet transmission of the Big-
`BrotherTM format, switching occurs on the video signal with—
`out need for the audio signal to be affected. Still further, as it
`will be clear from the following detailed description, switch-
`ing can also occur on the audio signal without need for the
`video signal to be affected.
`In the present specification. the output of the audio and
`video sources 12 will beusually called “audio file” and “video
`file”. IIowever, also a live encoded feed output is possible.
`The person skilled in the art will recognize that the particular
`kind of output from the sources 12 is not essential to the
`present invention, so that sometimes also the generic term
`“audio signal” and “video signal” will be used.
`
`CST000840
`
`CST000840
`
`7
`
`

`

`US 8,156,236 B2
`
`5
`The present invention will now be disclosed with reference
`to FIG. 3, which describes in greater detail the diagram shown
`in FIG. 2. First, the general operation ofthe system according
`to the present invention will be described with reference to
`three main events: 1) Request ofevent parameters; 2) Stream-
`ing; and 3) Switching. Subsequently, the software procedures
`adopted by the system according to the present invention will
`be described in a more detailed mamler.
`Request of Event Parameters
`The GUI manager 14 comprises a software procedure 22
`called interface builder. A first task of the interface builder 22
`is that of building a graphical representation of the event
`parameters, by requesting such parameters to the server, The
`request of parameters to the server is effected through a
`remote procedure call (RPC), using a client proxy 21. A client
`proxy, known as such,
`is a software object encapsulating
`remote procedure calls. The client proxy 21 commtmicates
`with a server stub 30, located on the server side. A server stub
`is known as such and its function is substantially specular to
`that of a client proxy. The event parameters requested by the
`interface builder 22 are accessed by the theatre descriptor 28.
`The theatre descriptor 28 is a software obj ect activated by the
`request of the interface builder 22, which operates by reading
`event information from a database on the server (not shown in
`the figures) and returning the event parameters to the client.
`Streaming
`As soon as the event parameters are returned to the client,
`the interface builder 22 requests the server to start streaming,
`the initial point ofView being a predefined point ofView ofthe
`event. In this respect, the interface builder 22 activates a
`further software procedure 26 on the server side, called ses-
`sion manager. The session manager 26 first reads the audio
`and video files to be streamed, by creating a stream reading
`procedure 40, called stream reader. The stream reader 40
`receives the outputs of the audio-video files 12 and preloads
`audio and video samples from each point of View in corre-
`sponding vectors. Once the audio and video samples are
`ready to be streamed to the client, the session manager 26
`generates a stream producer 34. The stream producer 34 is a
`software procedure responsible for performing a streaming
`session on the server side. More specifically, the stream pro-
`ducer 34 has the task of establishing a persistent connection
`with the client, sending stream global parameters to the client,
`and then sending the audio and video samples to the client.
`On the client side, the interface builder 22 creates a stream
`consumer 36 and a stream renderer 37. The stream consumer
`36 will receive samples from the stream producer 34, while
`the stream renderer 37 will render both the audio and the
`video streams. The GUI manager 14 also comprises an inter-
`face renderer 24, for rendering the user interface. More spe-
`cifically, the interface renderer 24 provides an abstraction
`layer which takes care ofdetails such as the operating system,
`the windowing interface, and the container application, like
`for example a Web browser, Should this be the case, the user
`could receive multimedia and interactive information inside
`the browser window at the same time as he is receiving the
`streaming data. The interface renderer 24 receives instruc-
`tions to render the specific user interface by means of a local
`method call.
`Switching
`As a consequence of what described above, the user can
`enjoy the event on the video window 50. The user can now
`switch from the current point of view to a different point of
`view by interacting, for example with the click of a mouse
`button, with active icons representing altemative points of
`view. These icons are shown as elements 11 .
`.
`. In in the GUI
`15 of FIG. 3. As soon as the user sends a switching request, a
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`4o
`
`45
`
`50
`
`55
`
`6O
`
`65
`
`6
`method of the user event manager 23 is activated. The user
`event manager 23 is a software object which is operating
`system dependent. The switching request is sent from the user
`event manager 23 to the server session manager 26, and from
`the server session manager 26 to the stream reader 40. The
`stream reader 40 does not alter the streaming of the audio
`samples along connection 19, but activates the streaming of a
`different video sample, corresponding to the requested point
`ofView. In order to minimize the loss of quality when switch-
`ing between video files, the switching preferably occurs when
`a key frame of the video samples corresponding to the
`requested point of view is encountered, as later explained in
`better detail. As soon as such key frame is encountered, the
`new point of View is streamed to the client.
`Consequently, even when switching between different
`points of view, the bandwidth of the streaming connection
`operated by the present invention, i.e. the network connection
`18 between the stream producer 34 on the server side and the
`stream consumer 36 on the client side, is the average band—
`width of a single audio/video stream, and not the cumulative
`bandwidth ofthe n audio/video streams, one for each point of
`view, as in the prior art systems of FIG. 1.
`The preferred embodiment of the present invention con-
`siders the case in which a single audio file and a plurality of
`video files, each video file representing a distinct point of
`view, are provided. However, different embodiments are also
`possible where a singlevideo file and a plurality ofaudio files,
`each audio file representing a different point of listening or a
`different audio source, are provided. Finally, also an embodi—
`ment with plural audio files and plural video files is possible.
`In the case ofa single video file and a plurality of audio files,
`switching between audio files will occur without altering the
`streamed video file. In the case of multiple video files and
`multiple audio files, switching will occur either on video files
`without altering the streamed audio file, or on audio files
`without altering the streamed video file. Should also the audio
`frames be provided with a key-frame technology, the audio
`switching preferably occurs when an audio key frame is
`encountered.
`The system according to the present invention is a distrib-
`uted application. A first way of implementing the system
`according to the invention provides for personal computers
`on the client side and two server stations on the server side. the
`first server station comprising the streaming server 11 and the
`second server station comprising the feed distributor 13. A
`second way provides forpersonal computers on the client side
`and one server station on the server side, the latter comprising
`both the streaming server 11 and the feed distributor 13. In
`this, way installation and maintenance of the system are
`easier and the communication time (latency) between the
`streaming server and the streaming distributor is reduced. A
`third way provides for both the client and the server residing
`on the same machine. A first example ofthis last embodiment
`is when the contents are distributed by means of a medium
`like a CD-ROM, where the use of a single machine is pre-
`ferred. A second example is when the contents are distributed
`in a place like an opera theatre, where each spectator is
`provided with an interactive terminal, used nowadays to allow
`the spectator to choose the captioning for the performance he
`is viewing in that moment, as adopted, for example, by the
`Metropolitan Theatre in New York. In that case, each specta-
`tor would be provided with a simple graphic interface (thin
`client), and the bulk of the system would reside on a single
`machine, for example a multiprocessor server with a UnixTM
`operating system. By managing different cameras, the spec—
`tator could use the present invention like some sort of “elec-
`tronic opera glass”.
`
`CST000841
`
`CST000841
`
`8
`
`

`

`US 8,156,236 B2
`
`7
`The preferred embodiment of the present invention is
`described with reference to a single server computer and to a
`single client operating in a WindowsTM environment, where
`the single client is representative of 11 different clients which
`can be connected to the server. The client computer can, for
`example, be a Pentium IIITM, 128 MB RAM, with a Windows
`9STM operating system. The server computer can,
`for
`example, be a Pentium IIITM, 512 MB RAM, with a Windows
`2000 ServerTM operating system. Visualization can occur on a
`computer monitor, a television set connected to a computer, a
`projection TV or visualization peripherals such as the PC
`GlasstronTM by Sony.
`Data streaming services can adopt a unicasting model or a
`multicasting model. In the unicasting model, every recipient
`is sent his own stream ofdata. A unique session is established
`between the unique IP address of the server and the unique IP
`address of the client. In the multicasting model, one single
`stream of data reaches the various users through routers,
`There is a single broadcast IP address for the server, which is
`used as a source of data for the different IP addresses of the
`various clients. However, in the current implementation over
`the Internet, routers first ignore and then discard multicast
`packets. Typically, routers are not configured to forward mul-
`ticast packets. As a consequence, the present invention pref—
`erably embodies a unicasting model. Moreover. the waste of
`bandwidth of the unicast method, i.e. multiple copies of the
`same data one for each client, is here an advantage because
`each client can personalize his or her own show.
`Advantageously, in the present invention, a particular user
`can control the switching between points ofview or between
`listening points for a number of other user. Further, it is also
`possible for switching commands to be preprogrammed, so
`that a switching between points of view or listening points
`occurs automatically, unless differently operated by the user.
`The operation ofthe system according to the present inven-
`tion will be now described in greater detail.
`Request of Event Parameters
`As soon as a client application is started, a specific event is
`requested. "his request can, for example, occur through a
`specific command line argument. From the point of View of
`the client application, an event is preferably described by the
`following event parameters:
`1) A number n of different points of view of the event;
`2) Textual description of each point of view;
`3) Logic identifier of each point of view, which is unique and
`preferably locally defined;
`4) Size (width and height) ofthe main window visualizing the
`current point of View;
`5) Stream bandwidth;
`6) Duration of the event; and
`7) Default (initial) point of view.
`These parameters are used by the client application to build
`the user interface for the requested event. More specifically,
`the client application should build:
`a) the correctly sized window 50 for the stream rendering, in
`accordance with parameter 4) above;
`. In of the GUI 15, each
`b) the n active (clickable) icons Il .
`.
`corresponding to a different point ofview, in accordance with
`parameter 1) above. Each of the icons ll
`.
`.
`. In will be
`correctly labeled in accordance with parameter 2) above; and
`c) a time indicator, which indicates the time elapsed com-
`pared to the total time,
`in accordance with parameter 6)
`above.
`Parameters 3), 5), and 7) will be stored for future use, later
`described in better detail.
`As already explained above, the interface builder 22 is a
`software object whose task is that of building the above
`
`
`
`5
`
`10
`
`15
`
`25
`
`30
`
`35
`
`4o
`
`45
`
`6O
`
`65
`
`8
`parameters. A C++ language definition of the interface
`builder 22 (CInterfaceBuilder) is, for example, the following:
`
`class CInterfaceBuilder {
`public:
`
`void BuildInterface(_'long int eventId);
`
`1;
`
`
`
`Throughout the present specification, the C++ program—
`ming language will be used to describe the functions, proce—
`dures and routines according to the present invention. Of
`course other programming languages could be used, like for
`example C, Java, Pascal, or Basic.
`In order to build the above parameters on the client side, the
`interface builder 22 will request such parameters to the server,
`by means of a remote procedure call RPC). A remote proce-
`dure call is sometimes also kno