US007139813B1
`
`(12)
`
`United States Patent
`Wallenius
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 7,139,813 B1
`Nov. 21, 2006
`
`(54) TIMEDEPENDENT HYPERLINK SYSTEM IN
`VIDEOCONTENT
`
`3/2002 Srinivasan et a1. ......... .. 725/32
`6,357,042 B1 *
`6,415,327 B1* 7/2002 Beckerman et a1.
`
`FOREIGN PATENT DOCUMENTS
`WO 98/44435
`4/1997
`
`W0
`
`OTHER PUBLICATIONS
`
`“Multimedia Hypervideo Links for Full Motion Videos”, IBM
`Technical Disclosure Bulletin. vol. 37, No. 04A, Apr. 1994, p. 95.
`XP-000986761 Hyperlinked Video,Jonathan Dakss, Stefan
`Agamanolis, Edmond Chalom, and V. Michael Bove, Jr. MIT Media
`Laboratory 20 Ames Street, Cambridge, Massachusetts 02139 USA
`
`p. 2-10.
`XP-000754600 VideoProXy: A Media and Protocol Converter for
`Internet Video, Kiyokuni Kawachiya Nagatsugu Yamanouchi
`Takayuki Kushida IBM Research, Tokyo research Laboratory,
`1623-14, Shimotsuruma, Yamato Kanagawa 242 Japan Jan. 10,
`1996 p. 541-550.
`* Cited b examiner
`y
`Primary Examinerilason cardone
`Assistant ExamineriAdnan M. MirZa
`(74) Attorney, Agent, or F irm4Cohen, Pontani, Lieberman
`& Pavane
`
`(57)
`
`ABSTRACT
`
`(75) Inventor: Jukka Wallenius, Helsinki (FI)
`
`(73) Assignee: Nokia Networks Oy, Espoo (FI)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U_S_C_ 154(1)) by 0 days_
`
`.
`(21) Appl' NO" 09/431,753
`
`.
`_
`(22) Flled'
`
`NOV‘ 1’ 1999
`
`(51) Int‘ Cl‘
`(200601)
`G06F 15/16
`(52) US. Cl. ....................... .. 709/219; 725/215; 725/22
`(58) Field of Classi?cation Search .............. .. 709/231,
`709/227, 245, 228, 203, 219; 725/22, 215
`See application ?le for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`5,572,442 A * 11/1996 Schulhof et al. .......... .. 364/514
`5,708,845 A
`1/1998 Wistendahl et al.
`5,774,664 A *
`6/1998 Hid
`et a1. ............. .. 395/200
`
`6/1999 Astgrzt al‘
`5,918,012 A
`5,961,603 A 10/1999 Kunkel et a1‘
`5,983,262 A * 11/1999 Kukkal ..................... .. 709/204
`5,996,015 A * 11/ 1999 Day et a1.
`6,012,086 A *
`1/ 2000 Lowell
`6,049,831 A :
`4/2000 Gardell et al. ............ .. 709/236
`6,141,693 A 10/2000 Perlman et a1‘
`6,195,692 B1 *
`2/2001 Hsu ......................... .. 709/219
`6,286,139 B1 *
`9/2001 Decinque . . . . .
`. . . . . . .. 725/5
`6,317,795 B1* 11/2001 Malkin et al. ..
`709/246
`6,330,595 B1 * 12/2001 Ullman et al. ............ .. 709/219
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`A t1me-dependent hyperlmk 1s associated W1th a v1deo of a
`content server. The t1me dependent lmk 1s cross-referenced
`with coordinates and time that it appears in the videoAuser
`may select the time-dependent hyperlink by selecting the
`hyperlink When is displayed by the video. The coordinates of
`the Selection, including time, are used to determine the
`hyperlink that Was selected. The broWser being used can
`then be Switched to a new Session indicated‘) the h erlink
`y
`yp
`'
`
`1110
`
`/
`-
`BROWSER
`
`100
`/
`
`BROWSER
`
`114
`
`/"°
`/
`
`112
`
`300
`/
`CPS
`
`200
`
`/
`C NTENT
`SCIJERVER
`
`jlo
`110-
`/ CONTENT
`4 /
`SERVER
`112'
`
`CONTENT
`SERVER \ 202
`
`64 Claims, 8 Drawing Sheets
`
`100
`
`/
`BROWSER
`
`300
`
`/
`CPS
`
`200
`
`/
`CONTENT
`SERVER
`
`MEDIA
`PROXY
`\ 400
`
`SERVICE CONTROL
`500
`MEANS
`SWITCHOVER -4
`APPLICATION
`
`100
`\
`BROWSER
`
`300
`200
`/ i
`CONTENT
`SERVER
`
`CPS
`
`MEDIA
`PROXY
`
`Hulu
`Exhibit 1005
`Page 0001
`
`

`
`U.S. Patent
`
`Nov. 21, 2006
`
`Sheet 1 0f 8
`
`US 7,139,813 B1
`
`INPUT/OUTPUT
`DEVICE
`
`100
`\
`
`150~
`
`D'SPLAY
`
`BROWSER
`
`110
`\
`
`f 200
`1ST CONTENT
`SERVER
`
`16%,. INPUT //
`DEV‘CE
`
`‘
`
`2ND CONTENT j/
`SERVER
`
`202
`
`140
`
`FIG. 1a
`
`Hulu
`Exhibit 1005
`Page 0002
`
`

`
`U.S. Patent
`
`Nov. 21, 2006
`
`Sheet 2 of8
`
`US 7,139,813 B1
`
`[~15
`
`M30
`
`ASSOCIATE
`HYPERLINK
`WNIDEO CONTENT
`IL
`DISPLAY VIDEO
`CONTENT
`1
`SELECT HYPERLINK
`I
`DETERMINE
`SESSION
`DESCRIPTION
`INFORMATION
`I
`INITIATE
`CONNECTION TO
`NEW LINK
`I
`SWITCH
`CONNECTION TO
`NEW LINK
`
`in
`
`FIG. 1b
`
`Hulu
`Exhibit 1005
`Page 0003
`
`

`
`U.S. Patent
`
`Nov. 21, 2006
`
`Sheet 3 0f 8
`
`US 7,139,813 B1
`
`TRANSMIT CONTENT
`REFERENCE BEFORE
`HYPERLINK VIEWABLE
`
`I
`
`INITIATE CONNECTION TO
`N40
`NEW LINK
`
`NO
`
`SELECT
`HYPERLINK?
`
`20
`
`DISCONNECT
`NEW SESSION
`
`SWITCH
`CONNECTION TO
`NEW LINK
`
`FIG. 10
`
`Hulu
`Exhibit 1005
`Page 0004
`
`

`
`U.S. Patent
`
`Nov. 21, 2006
`
`Sheet 4 0f 8
`
`US 7,139,813 B1
`
`M70
`
`Ci)
`
`DOWNLOAD
`SWITCHOVER
`APPLICATION
`I
`ESTABLISH NEW
`SESSION VIA
`SWITCHOVER
`APPLICATION
`I
`PAUSE CONTENT
`STREAM OF ORIGINAL
`CONTENT AT
`SWITCHOVER
`I
`RESUME ORIGINAL
`CONTENT AT
`SWITCHOVER BACK TO
`ORIGINAL CONTENT
`I
`RELEASE NEW
`SESSION
`ESTABLISHED
`IN STEP 72
`
`@
`
`FIG. 1d
`
`Hulu
`Exhibit 1005
`Page 0005
`
`

`
`U.S. Patent
`
`Nov. 21, 2006
`
`Sheet 5 0f 8
`
`US 7,139,813 B1
`
`100
`
`/
`
`‘
`~
`BROWSER ‘
`
`110
`
`/
`/
`
`112
`
`FIG. 2a
`
`/2OO
`
`CONTENT
`SERVER
`
`100
`/
`
`BROWSER
`
`114
`
`300
`/
`
`CPS
`
`<
`:
`
`200
`/
`
`CONTENT
`SERVER
`
`30'
`
`/
`112'
`
`2ND
`CONTENT \
`SERVER
`202
`
`FIG. 2b
`
`Hulu
`Exhibit 1005
`Page 0006
`
`

`
`U.S. Patent
`
`NOV. 21, 2006
`
`Sheet 6 0f 8
`
`US 7,139,813 B1
`
`/100
`
`300
`
`BROWSER
`
`CPS
`
`200
`CONTENT
`SERVER
`
`\ MEDIA
`
`FIG. 2C
`
`PROXY
`\4o0
`
`SERVICE CONTROL *
`MEANS
`SWITCHOVER
`APPLICATION
`
`0
`5 0
`
`100
`
`‘
`
`300
`
`200
`
`BROWSER
`
`CPS
`
`CONTENT
`SERVER
`
`MEDIA
`PROXY
`
`FIG. 20'
`
`400
`
`Hulu
`Exhibit 1005
`Page 0007
`
`

`
`U.S. Patent
`
`Nov. 21, 2006
`
`Sheet 7 0f 8
`
`US 7,139,813 B1
`
`100
`
`BROWSER
`
`200
`/
`CONTENT
`SERVER
`
`DATABASE
`
`_ L (
`250
`
`FIG. 3
`
`Hulu
`Exhibit 1005
`Page 0008
`
`

`
`U.S. Patent
`
`Nov. 21, 2006
`
`Sheet 8 0f 8
`
`US 7,139,813 B1
`
`Hulu
`Exhibit 1005
`Page 0009
`
`

`
`US 7,139,813 B1
`
`1
`TIMEDEPENDENT HYPERLINK SYSTEM IN
`VIDEOCONTENT
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`The present invention relates to a hyperlink having a time
`dependency and related to a video.
`2. Description of the Related Art
`Hypertext systems, such as the World Wide Web (WWW),
`contain objects, i.e. electronic documents, Which are cre
`atively linked to each other. During a session, a user device
`is connected via a broWser to a content server including an
`object and content is exchanged betWeen the user device and
`the content server. The exchange of content occurs over the
`internet using, for example, a hypertext transport protocol
`(HTTP) for hypertext documents. The HTTP protocol mes
`sages are carried in Transmission Control Protocol/Internet
`Protocol (TCP/IP) messages. It is also possible to transmit
`voice or video content over the internet. For the establish
`ment of voice and/or video multimedia session protocols
`like Session Initiation Protocol (SIP) and H.323 are used.
`Multimedia sessions can be established via SIP by selecting
`links from HTTP hypertext documents, if the broWser is
`equipped With multimedia capabilities. SIP is a client-server
`protocol in Which requests are generated by one entity (the
`user or client) and sent to a receiving entity (the server).
`Contacting the server of the neW object may be accom
`plished by looking up the domain name of the neW server in
`a Domain Name Service (DNS) and determining the correct
`IP address. The DNS is actually its oWn netWork. If one DNS
`server does not knoW hoW to translate a domain name, it asks
`another, and so one, until the correct IP address is found. The
`request from the broWser is sent to the IP address and may
`be redirected to Where the actual object is connected. The
`object returns a response through the chain.
`Each object Which the user device may access may
`include one or more hyperlinks that link to another place in
`the same document or to an entirely different object. The
`hyperlinks occur Within a certain area in the object and may
`include a picture or phrase, a Word, an icon, or any other
`visual representations. The hyperlinks in these objects may
`be selected by clicking When the cursor is Within the area of
`the hyperlink. When the hyperlink is selected, the broWser of
`the user device sWitches over to a neW object that is
`associated With the hyperlink.
`Prior art video content include hyperlinks and other
`WWW type content associated With video programs. HoW
`ever, the hyperlinks and WWW type content that are asso
`ciated With prior art video content are transmitted in parallel
`to the video program Without being associated directly to
`spatial objects Within the video stream. Therefore, these
`hyperlinks are not bound to objects in the video. It is
`desireable to navigate in video/multimedia content using
`hyperlinks bound to the video content. Similarly, the prior
`art does not de?ne mechanisms for implementing a seamless
`sWitchover betWeen video contents. In the prior art, there is
`alWays an observable delay When establishing a neW session
`after a hyperlink has been selected from, e.g., a Web page
`displayed by the side of a television picture.
`Furthermore, in the prior art, there are no mechanisms
`having parallel multimedia streams active for potential
`sWitchover periods While retaining only one multimedia
`stream toWard the user device.
`
`20
`
`25
`
`30
`
`35
`
`40
`
`50
`
`55
`
`60
`
`65
`
`2
`SUMMARY OF THE INVENTION
`
`It is an object of the present invention to provide a system
`in Which link information is associated With video such that
`a user may select a hyperlink in video/multimedia content.
`According to the invention, a user of a video broWser
`selects a link displayed With the video While vieWing the
`video. Once the link to a neW session is selected, the
`establishment of a connection to the neW session is initiated.
`The links may be represented by three dimensions including
`the x and y coordinate ranges of the link at a given moment
`in time in the video image With the last dimension in each
`case being time that the link appears during the playing of
`the video. For example, the video may shoW a person
`Walking from room to room and the hyperlinks may be
`associated With different furniture in each of the rooms.
`Therefore, the hyperlink for a chair in the kitchen is only
`selectable for the portion of time that the chair is displayed.
`The link may be to a site containing information about the
`history of the fumiture (i.e. if the fumiture is an antique)
`and/or information about hoW to buy the furniture. Further
`more, it is conceivable that the hyperlink may be represented
`in three dimensional space With a fourth dimension being
`time if the video is in three dimensions.
`The link information may be transmitted as a link stream
`to the video broWser parallel to the video information so that
`the link information is available to the video broWser When
`the link is in effect. In this embodiment the establishment of
`a connection to a neW session is made by the video broWser.
`The link information may alternatively be stored in the
`content server in a database With a mapping of time and
`coordinates to the link information. In this embodiment, the
`establishment of a connection to a neW session may be made
`by the content server for the video broWser.
`In another embodiment, a call processing server may be
`arranged betWeen the content server and the broWser. In this
`embodiment, the call processing server may be used to
`initiate the establishment of a neW session, thereby reducing
`bandWidth required to the broWser. Moreover, a media proxy
`arrangement may optionally be connected With the call
`processing server.
`In yet another embodiment, the link information may be
`doWnloaded to the video broWser from the content server at
`the start of the video session before the video information is
`transmitted to the video broWser from the content server. In
`this embodiment, the mapping of time and coordinates to the
`link information occurs in the video broWser.
`Each of these embodiments may include the advance
`establishment of a connection to a neW session of an
`approaching hyperlink to provide a smooth sWitchover to the
`neW session should the approaching hyperlink be selected.
`Furthermore, a sWitchover application may be used to
`control the sWitchover from the original session and content
`stream to a neW session and control stream to Which a
`selected hyperlink in the original content stream points. The
`sWitchover application may be doWnloaded from the origi
`nal content server to the broWser, the call processing server,
`or a service control means associated With the call process
`ing server. The sWitchover application may be adapted to the
`applications required by the content With Which it is asso
`ciated.
`The various features of novelty Which characteriZe the
`invention are pointed out With particularity in the claims
`annexed to and forming a part of the disclosure. For a better
`understanding of the invention, its operating advantages, and
`speci?c objects attained by its use, reference should be had
`
`Hulu
`Exhibit 1005
`Page 0010
`
`

`
`US 7,139,813 B1
`
`3
`to the drawing and descriptive matter in which there are
`illustrated and described preferred embodiments of the
`invention.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`In the drawings, wherein like reference characters denote
`similar elements throughout the several views:
`FIG. 1a is a basic schematic diagram showing a system
`for processing a time-dependent hyperlink according to the
`present invention;
`FIGS. lbild are ?ow diagrams showing the general steps
`for selecting a hyperlink within a video and transferring
`control between servers;
`FIG. 2a is a schematic diagram showing a ?rst embodi
`ment connection between a content server and a video
`browser of the present invention in which the link informa
`tion is transmitted to the video browser in a link stream;
`FIG. 2b is a schematic diagram of the connection of FIG.
`2a showing an optional call processing server;
`FIG. 20 is a schematic diagram of the connection of FIG.
`2a showing an optional media proxy with the call processing
`server;
`FIG. 2d is a schematic diagram of the connection of FIG.
`20 showing an optional service control means connected to
`the call processing server; and
`FIG. 3 is a schematic diagram showing a second embodi
`ment connection between a content server and a video
`browser of the present invention in which the link informa
`tion is stored in a link database in the content server; and
`FIG. 4 is a timing diagram showing the transmission of a
`switchover application and content stream from a content
`server to a browser.
`
`DETAILED DESCRIPTION OF THE
`PRESENTLY PREFERRED EMBODIMENTS
`
`Referring to FIG. 1a, a browser 100 is used with a user
`input/output device 140 including a display device 150 and
`an input device 160 such as, for example, a computer mouse
`and/or a keyboard. The browser 100 connects to a content
`server 200 to retrieve information from an electronic docu
`ment in the content server 200 via a channel 110 and display
`the information to the user. The session between the browser
`100 and the content server 200 is conducted over an IP
`network using a data transfer protocol such as, for example,
`a Real Time Transport Protocol (RTP) which separates the
`electronic document in media appropriate packets (i.e. a
`video frame) for transmission. The electronic document
`transmitted to the browser from the content server 200 may
`contain hyperlinks which link to other documents which
`may be in content server 200 or may be in other content
`servers. The link data is transmitted to the browser 100 using
`another protocol which includes more error checking, such
`as User Datagram Protocol/Intemet Protocol (UDP/IP) or
`Transmission Control Protocol/Intemet Protocol (TCP/IP).
`To select a hyperlink, the user operates the input device 160
`to select the position in the electronic document at which the
`hyperlink is shown (i.e., the x, y coordinates on the user
`display). According to the invention, the electronic docu
`ment in the content server 200 is a video and the hyperlinks
`are viewable objects in the video of the content server 200.
`Therefore, the position of the hyperlink also includes time as
`one of the coordinates. Content reference such as Uniform
`Resource Locator (URL) and/or session description infor
`mation of the selected hyperlink is determined from the
`coordinates selected by the user. If for example, the hyper
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`link is linked to an electronic document in content server
`202, the browser initiates a connection to a new session the
`electronic document in content server 202. To establish the
`new session, the new content server 202 must be contacted
`and a description of the session to be established and/or an
`identi?er to specify the content requested (URL) must be
`delivered to the new content server 202. These steps may be
`performed by a Session Initiation Protocol (SIP). The
`description of the new session to be established may be
`described in a Session Description Protocol (SDP). The
`SDPs may be multicast via a well-known multicast address.
`A more detailed description of the RTP is found in H.
`SchulZrinne, S. Casner, R. Fredrick, and V. Jaconson, “RTP:
`a transport protocol for real-time applications,” Request for
`Comments (Proposed Standard) 1889, Internet Engineering
`Task Force, January 1996, the entire contents of which are
`incorporated herein by reference. A more detailed descrip
`tion of SIP is found in DRAFT-IETF-MMUSIC-SIP at
`www.itf.org, the entire contents of which are incorporated
`herein by reference. Once the new content server is con
`nected, the browser switches the user input/output device
`140 to the new content server 202 and drops the connection
`to the content server 200.
`Referring to FIG. 1b, the present invention includes the
`association of hyperlinks with the video as three dimen
`sional objects where the dimension include the location on
`the screen (x, y) at which the hyperlink appears in the video
`and the time (t) during the playing of the video at which the
`hyperlink appears. It is also conceivable that the video may
`be three dimensional wherein the hyperlink has x, y, Z, and
`t coordinates. The ?rst step according to the invention is to
`associate the hyperlinks with a time and place in the video
`by coordinates, step 10. That is, each hyperlink is associated
`with coordinates indicating an area of a display and a time
`during the playing of a video at which the hyperlink appears
`at those coordinates. As a user is viewing the video via
`browser 100, the user may select one of the hyperlinks
`associated with the video, step 20. Upon selection of the
`hyperlink in the video by the user, a content reference for the
`selected hyperlink is determined based on the coordinates
`selected and the time of the selection, step 30. The content
`reference may include a Uniform Resource Locator (URL)
`to establish a session to the new content, session description
`information to re?ect the characteristics of the new content,
`and the requirements for the browser. The URL may also
`include a time displacement from the start of the new content
`at which to start playing the new content. For example, the
`hyperlink may point to a portion, such as a scene, of a video
`which is located in the middle of the video. The URL may
`also specify a limited area in the picture frames from the
`time displacement onward. The content server may focus the
`picture to display only this limited area. Alternatively, the
`showing of the limited area in the picture frames and the
`focussing as speci?ed by the URL may be performed by the
`browser itself, provided that the video coding of the stream
`from the content server has suf?cient precision. However,
`the time displacement may also be indicated otherwise. The
`content reference may further comprise an SIP Call Pro
`cessing Language (CPL) script that is sent in the session
`establishment signalling and used by call processing servers
`(described later) to assist in ?nding the required content. The
`CPL script may comprise contact instructions to alternate
`locations to be contacted to reach the required content.
`Similarly, the content reference may also comprise execut
`able content that speci?es how the selection of the hyperlink
`appears to the user. The content reference may include
`instructions to play a video clip at a hyperlink selection and
`
`Hulu
`Exhibit 1005
`Page 0011
`
`

`
`US 7,139,813 B1
`
`5
`control the establishment of the connection to the referred
`content. Finally, the content reference may specify Whether
`a smooth sWitchover or immediate sWitchover is required
`and at Which point in time the sWitchover should occur. After
`the content reference is determined, connection to the neW
`session is initiated, step 40. When the connection to the neW
`session is established, the broWser 100 begins the neW
`session and ends the previous session from Which the neW
`link Was selected, i.e. the broWser 100 sWitches over to the
`neW session, step 50.
`Regarding the dimension information (x-range, y-range,
`time) for the content, the x-range and y-range of a hyperlink
`at a given time WindoW (i.e., time range) may be provided
`explicitly in the link stream. Alternatively, an algorithm may
`be used by the broWser that selects an area on the screen
`based on visual characteristics of the image on the screen.
`The algorithm may comprise adaptive learning characteris
`tics to enable it to select objects of frequently occurring
`types of patterns or persons. An algorithm that tracks a
`moving object in the video stream for a speci?ed time may
`also be provided in the broWser. The starting coordinates,
`starting time, and duration that the tracking should be
`performed may be speci?ed to the tracking algorithm.
`The algorithm may be built into the broWser or may be
`doWnloaded to the broWser When the broWser initiates a
`session With the content that the algorithm is associated
`With. Similarly, the learning information may also be doWn
`loaded. The algorithm makes a decision Whether an object
`has been pointed to by a pointer device based on pointing
`coordinates and time.
`The existence of a link may be indicated to a vieWer of the
`video in various Ways. In one embodiment, the video content
`itself may contain information indicating visually to a user
`the existence of a hyperlink. In another embodiment, the
`broWser may display a visual object pointing to the video
`content having the object having the associated hyperlink. In
`yet another embodiment, the broWser may display the
`bounds of the object in accordance With the boundaries of
`the object provided explicitly in the link information con
`tained in the link stream from the content server.
`Referring to FIG. 2a, an embodiment of the invention is
`shoWn in Which a user vieWs the video using the video
`broWser 100 connected to the content server 200 Which
`contains the video to be played. In this embodiment, the
`video and content reference information for the hyperlinks
`in the video are transmitted in tWo separate streams on
`respective ?rst and second channels 110, 112. A link stream
`containing the content reference information for the hyper
`links is transmitted in parallel With the video to the video
`broWser 100. The link stream includes the information
`required to establish the link, i.e., content reference and
`information suf?cient to visualiZe the existence of the hyper
`link. In this embodiment, the link stream provides the
`content reference for the hyperlinks that are presently vieW
`able and selectable in the video. For example, the video may
`shoW a person Walking from room to room and the hyper
`links may be associated With different fumiture in each of
`the rooms. Therefore, the hyperlink for a chair in the kitchen
`is only selectable for the portion of time that the chair is
`displayed. Accordingly, the link stream contains content
`reference information for the hyperlink to the chair only for
`the duration of time that the chair is displayed.
`When the user makes a selection in step 20 of FIG. 1b, the
`video broWser 100 has all the information it needs to
`establish a connection, i.e. perform steps 30, 40, and 50 of
`FIG. 1b. Once the connection to the electronic document of
`the selected link is established, the broWser 100 begins a
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`session With the neW link. The electronic document to Which
`the hyperlink is linked may be any type of electronic
`document and does not have to be another video. The linked
`document could be a Web page, a hypertext document, a
`video clip, and audio clip, or any other type of electronic
`document.
`In an optional embodiment according to a How diagram of
`FIG. 10, the broWser may initiate the establishment of a
`session toWards the content pointed to by the link, i.e., the
`link session, before the user selects the hyperlink. In this
`embodiment, the content reference such as the address (i.e.,
`URL) toWards the content pointed to by the link and the link
`visualiZation information (coordinates and time ranges) for
`a hyperlink is transmitted to the broWser 100 before the
`hyperlink is vieWable in the video, step 30. The broWser then
`initiates the establishment of a connection to that hyperlink
`on a separate channel While the original content is still being
`vieWed by the user, step 40. Accordingly, When the user
`selects the hyperlink, step 22, a smooth sWitchover is
`achievable in step 50 because a parallel connection and
`optionally the content stream is already established by the
`time the hyperlink appears in the video and can be selected
`by the user. If the hyperlink is not selected, the broWser
`merely drops the neWly made connection and maintains the
`current session, step 52. This embodiment is costly and
`Wastes bandWidth. HoWever, it provides a smooth sWitcho
`ver from the current session to the neW session When the
`hyperlink to the neW session is selected. The smooth
`sWitchover is possible because the session establishment
`delay and the content loading delay at the content server are
`eliminated. Similarly, the delay of searching to the right
`position Within the content, eg in a CD-ROM device, is
`eliminated. In this embodiment, the content reference infor
`mation must be transmitted to the video broWser 100 by the
`link stream in advance of the actual video Which shoWs the
`hyperlink so that the advanced establishment of the link is
`possible. The advanced connection may be selectively made
`for those links Which are likely to be selected, or other
`criteria.
`Similarly, a smooth sWitchover can be used When retum
`ing to the original content from a selected content either
`When the content has reached its end or When a user
`explicitly decides to do so. When approaching the end of a
`branched-to content, the broWser 100 starts the reestablish
`ment of a session toWards the original content from Which
`the link Was selected. The procedure for reestablishment of
`the original content is similar to the establishment of an
`upcoming link. In some cases, if the content reference
`indicates that the link content of a selected link is suf?ciently
`short, the session to the original content server may be
`maintained at link selection. HoWever, When the link content
`of the selected link reaches its end, it may be necessary to
`request the original content server to restart transmitting the
`content. If the original content server does not maintain
`content state, a time displacement at the branched-from link
`is indicated to it.
`In a further optional embodiment shoWn in FIG. 2b, a call
`processing server 300 is arranged betWeen the broWser 100
`and the content server 200. The parallel ?rst and second
`transmission channels 110', 112' are arranged betWeen the
`content server 200 and the call processing server 300. When
`the user selects a hyperlink, the coordinates and time of the
`selection are transmitted to the call processing server 300
`from the broWser 100. The call processing server 300
`performs the steps 30 and 40 of FIG. 1b, thereby selecting
`the content reference and establishing the session, i.e.,
`setting up a connection to a neW content server 202 based on
`
`Hulu
`Exhibit 1005
`Page 0012
`
`

`
`US 7,139,813 B1
`
`20
`
`25
`
`30
`
`35
`
`7
`the coordinates and the time of the selection and initiating
`the connection to the neW content server 202 to Which the
`neW link is connected, over a neW channel 118. The call
`processing server 300 also establishes the neW session in
`step 50 of FIG. 1b by dropping the old session from the ?rst
`content server 200 and connecting the neW content server
`202 to the broWser 100, step 50 of FIG. 1, When the
`connection to the neW session is established. The call
`processing server 300 may establish this neW session in
`advance of the selection of the neW session by the user to
`enable a smooth transition. The bene?t of using the call
`processing server 300 is that only one session is connected
`to the video broWser 100 at the vieWer terminal 140 at any
`one time, Which may save bandWidth.
`In yet a further embodiment shoWn in FIG. 20, a media
`proxy 400 may optionally be attached to the CPS 300. In this
`embodiment, the control channels betWeen the broWser 100
`and the content server 200 are connected via the CPS 300.
`HoWever, the data channels carrying the individual media
`components are routed from the content server 200 to the
`broWser 100 via the media proxy 400. On this embodiment,
`the CPS 300 instructs the media proxy 400 to perform
`connections on the data channels. The data channels may
`comprises a media mix including audio, video, and data
`streams. The different media can be added, dropped, or
`replaced dynamically during a call via instructions from the
`CPS 300. The media proxy 400 identi?es the different types
`of media components from the multimedia stream and
`perform sWitching of the individual media components
`separately.
`In yet a further embodiment shoWn in FIG. 2d, a service
`control means 500 may be connected With the CPS 300. In
`this embodiment, control signals and the link stream are
`transmitted to the service control means 500 from the
`broWser 100 and the content server 200. The data streams are
`transmitted via a media proxy 400. If the media proxy 400
`is not used, the data stream is transmitted to the CPS 300
`directly.
`Each of the embodiments shoWn in FIGS. 2ai2d and
`discussed above may include the advanced connection of a
`link for a smooth sWitchover. Furthermore, a sWitchover
`application that performs the smooth sWitchover to a neW
`session may optionally be used. The sWitchover application
`is a generic program for performing several tasks and may
`be responsible for one to many links associated With a video,
`i.e., the sWitchover application is associated With the video
`content and knoWs the links it is responsible for. The
`sWitchover application is stored in the content server With
`the video.
`The sWitchover application may be doWnloaded (step 70,
`FIG. 1d) from the content server of the video to the broWser
`100 (FIG. 2a), the CPS 300 (FIGS. 2b and 20), or to the
`service control means 500 (FIG. 2d). The sWitchover appli
`cation is executed by the broWser 100, CPS 300, or service
`control means 500. The sWitchover application is respon
`sible for receiving and interpreting the link stream informa
`tion. The sWitchover application may establish the sessions
`toWard links selectable by a vieWer of the video. A link
`selection by a user is indicated to the sWitchover application
`from the broWser 100, and the sWitchover application hands
`control over from the current session to a neW session. A
`particular sWitchover application may include any one or
`more of the folloWing steps:
`DoWnloading given contents at given points in time;
`Playing contents at given points in time;
`Checking content server availability;
`
`40
`
`50
`
`55
`
`60
`
`65
`
`8
`Selecting from several content servers depending on
`server load status;
`Requesting a content server to prepare a content for
`transmission (e.g., load from a secondary storage to a
`buffer in case of failure in one or more of the bulleted
`steps listed here);
`Requesting a content server to start transmitting the
`prepared document;
`Requesting a content server to start transmitting content
`from a given time displacement;
`Requesting the doWnloading of a neW sWitchover appli
`cation from a content server for the content referred to
`by a selected link;
`Establishing sessions to neW content at given points in
`time;
`Determining round trip delay betWeen the broWser and the
`content server and adjusting sWitchover times on the
`determination;
`Determining permanent terminal or terminal location spe
`ci?c implications to general session establishment time
`and adjusting sWitchover schedules based on the deter
`mination (e