Ulllted States Patent [19]
`Cannon et al.
`
`[54] METHODS AND APPARATUS FOR
`IMPLEMENTING CONTROL FUNCTIONS IN
`A STREAMED VIDEO DISPLAY SYSTEM
`
`[75] Inventorsl Anthony CaHI10I1;David del Val, both
`of Mountain View; Anders Klemets,
`Sunnyvale, an of Calif
`
`.
`.
`_
`[73] Ass1gnee: M1cr0s0ft C0rp0rat10n, Redmond,
`Wash.
`
`[21] Appl- NO-I 08/819,586
`
`US006014706A
`[11] Patent Number:
`[45] Date of Patent:
`
`6,014,706
`Jan. 11, 2000
`
`5,625,405
`5,717,691
`
`4/1997
`5/
`41998
`..
`57687533 6 1998
`578287848 10/1998 MacCormack etal. .............. .. 709/247
`7
`7
`
`FOREIGN PATENT DOCUMENTS
`
`0605115 7/1994 European Pat. Off. .
`0653884 5/1995 European pat Off_ _
`0676898 10/1995 European Pat. Off. .
`0746158 12/1996 European Pat. Off. .
`OTHER PUBLICATIONS
`
`[22] Filed:
`
`Mal? 14, 1997
`_
`_
`Related U-S- Appllcatlon Data
`[60] Provisional application No. 60/036,662, Jan. 30, 1997, and
`provisional application No. 60/036,661, Jan. 30, 1997.
`
`Chen, H.J., et al., “A Scalable Video—on—Demand Service
`for the Provision of VCR—Like Functions”, IEEE Proceed
`ings of the International Conference on Multimedia Com
`paling and Systems, Washington, DC, 65—72, (May 15—18,
`1995)
`
`[51]
`
`Int. Cl.7 .................................................... .. G06F 13/00
`U-S- Cl- ........................ ..
`
`709/248
`[58] Field of Search ....................... .. 395/20048, 200.61,
`395/20063, 200.59, 200.65, 182.16; 709/217,
`218, 219, 229, 231, 233, 235, 24s; 371/30,
`32> 33; 707/10
`
`_
`References Clted
`U S PATENT DOCUMENTS
`'
`'
`6/1990 Isle et al. .............................. .. 364/513
`9/ 1991 Golestani - - - - - -
`- - - - -- 370/60
`6/1992 Beltel ct a1~ ~~
`395/154
`5/
`geltefl ,et al'l "" "
`8;1994 Glélsrtrllgileettaai
`5/1995 Hooper et aL'
`5:414:455
`7/1995 Chimento’ Jr” et aL _
`574347848
`5,455,910 10/1995 Johnson et a1, _____ __
`5,481,542
`1/1996 Logston et al.
`5,490,252
`2/1996 Macera et al.
`5,504,744
`4/1996 Adams et a1~
`575197701
`5/1996 Colmant et al' "
`gilaerrlldzttaill'etgl ' ' '
`7/1996 Branstad et a1‘
`
`[56]
`
`4,931,950
`5,050,161
`5,119,474
`5,274,758
`
`5:537j4O8
`
`"'"395/2/OO
`348/7
`__ 370/17
`395/650
`.. 370/94.2
`395/200-1
`~~ 370/601
`" 370/601
`' '
`I
`370/7'9
`
`Primary Examiner_prank J_ Asta
`Assistant Examiner_Jason
`Cardone
`Attorney, Agent,
`or Firm—ScheWgman, Lundberg,
`Woesnner & Kluth, P.A.
`
`ABSTRACT
`[57]
`A method for displaying streamed digital video data on a
`client computer. The client computer is con?gured to receive
`the streamed digital video data from a server computer via
`a computer network. The streamed digital video data is
`transmitted from the server computer to the client computer
`as a stream of video frames. The method includes receiving
`a ?rst plurality of video frames at the client computer. The
`plurality of video frames represents a subset of the stream of
`video frames. The stream of video frames comprises inde
`pendent playable video frames and dependent playable
`video frames. The‘ method further includes'displaying'the
`?rst plurality of video frames on a video display terminal
`associated With the client computer. There Is further
`included issuing a reWind command from the client com
`puter to the server. The reWind command causes a second
`plurality of video frames of the stream of video frames
`different from the ?rst plurality of video frames to be
`streamed from the server computer to the client computer.
`ifhe second pll?ralilty of video frames has been streamed at
`
`east Once to t e C lent Commit“
`
`28 Claims, 8 Drawing Sheets
`
`. . . . .. 370/84
`5,566,175 10/1996 Davis . . . . . . . . . . . .
`.. 370/681
`5,574,724 11/1996 Bales et al.
`5,623,690
`4/1997 Palmer et al. ........................ .. 395/806
`
`110
`
`ENCODER
`
`CLIENT PLAY-OUT
`BUFFER
`
`PAGE 1 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 1 0f 8
`
`6,014,706
`
`110
`s
`
`A
`
`ENCODER
`
`115
`
`106
`
`100
`
`11
`
`v
`
`11
`
`s
`\l 0o 0
`
`o:
`
`10
`
`A
`
`~102
`
`SERVER
`
`N u 45=
`
`(
`1 18 RETRANSMISSION
`BUFFER
`
`~114
`
`~116
`
`CLIENT
`
`120
`
`122
`"
`
`= RENDERER
`
`CLIENT PLAY-OUT
`BUFFER
`
`FIG. 1A
`
`PAGE 2 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 2 0f 8
`
`6,014,706
`
`Kmrzmd
`
`331
`@8
`wnwl
`3% P
`%
`
`mop
`
`3K9
`339 l}
`3%:
`
`\/
`
`PAGE 3 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 3 0f 8
`
`6,014,706
`
`oowi_l\ll
`
`ll
`
`_
`
`_ _
`
`u @R
`
`_ ..
`
`
`
`r l l I l I l I l I l I I l I I l I I l l l I I I .w.
`
`NPN
`
`OK
`
`1k
`
`wow)
`
`00
`
`A
`_ _ _ V
`
`
`
`_ \ mam \ V
`
`
`
` m \ 8N) om! “ 8N;
`
`A
`
`E052
`
`mam
`
`M
`
`V
`
`oNN
`
`NNN
`
`..
`
`201
`
`PAGE 4 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 4 0f 8
`
`6,014,706
`
`
`
`own
`
`3n Non mwn own wwn N N N a g 5
`
`00m
`
`n .o_|._
`
`PAGE 5 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`U.S. Patent
`
`Jan. 11, 2000
`
`~GEEPEaaeaeae
`
`
`
`bevCovObBlvQIvvlyZlvOl”80KYOHOWZOoop
`
`Sheet 5 of 8
`
`6,014,706
`
`y‘Sls
`
`Q9r99F79%ZIPOOFBSrOSr+SrzorOS”HOEo!ooeB
`
`PAGE 6 of21
`
`PETITIONERS' EXHIBIT 1009
`
`PAGE 6 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`U.S. Patent
`
`Jan. 11, 2000
`
`Sheet 6 0f 8
`
`6,014,706
`
`500
`
`CLIENT SENDS COMMAND
`AND TIME PARAMETER
`
`I
`
`SERVER ASCERTAINS FIRST
`DATA PACKET TO SEND
`
`I
`
`SERVER AND CLIENT
`FLUSH BUFFERS
`
`I
`
`SERVER COMMUNICATES TO
`CLIENT FIRST PACKET
`SEQUENCE NUMBER
`
`I
`
`SERVER STREAMS PACKETS
`TO CLIENT
`
`I
`
`CLIENT DISPLAYS
`STREAMED DATA PACKETS
`
`~ 502
`
`~ 504
`
`~506
`
`~ 508
`
`~512
`
`514
`
`FIG. 5
`
`PAGE 7 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 7 0f 8
`
`6,014,706
`
`530
`
`504 ® /
`
`SERVER SEEKS BACKWARD/FORWARD
`FOR SEEKABLE FRAME IMMEDTATELY
`NEXT TO FRAME ASSOCIATED
`WITH TIME PARAMETER OF
`STEP 502
`
`V
`DESIGNATE SEEKABLE FRAME
`OF STEP 532 FIRST FRAME
`TO SEND TO CLIENT
`
`FIG. 6 @536
`
`@550 504
`
`SERVER WAITS FOR NEXT
`SEEKABLE FRAME FROM
`ENCODER
`
`DESIGNATE SEEKABLE FRAME
`OF STEP 552 FIRST FRAME
`TO SEND TO CLIENT
`
`@5556
`
`FIG. 7
`
`~ 532
`
`~ 534
`
`~ 552
`
`~ 554
`
`PAGE 8 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 8 0f 8
`
`6,014,706
`
`@800
`
`SERVER SEEKS BACK IN FAST FORWARD 802
`STREAM FOR FIRST SEEKABLE FRAME "'
`STARTING FROM FRAME WHOSE TIME
`STAMP MOST CLOSELY CORRESPONDS TO
`TIME STAMP OF STEP 502
`
`804
`”
`
`II
`DESIGNATE SEEKABLE FRAME
`OF STEP 802 FIRST FRAME
`TO SEND TO CLIENT
`
`FIG. 8 @806
`@900
`
`SERVER SEEKS BACK IN PLAY
`STREAM FoR FIRST SEEKABLE FRAME “902
`STARTING FRoM FRAME WHOSE TIME
`STAMP M0ST CLOSELY CORRESPONDS T0
`TIME STAMP 0F STEP 502
`
`II
`DESIGNATE SEEKABLE FRAME
`OF STEP 902 FIRST FRAME
`TO SEND TO CLIENT
`
`904
`"'
`
`@806
`
`FIG. 9
`
`PAGE 9 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`1
`METHODS AND APPARATUS FOR
`IMPLEMENTING CONTROL FUNCTIONS IN
`A STREAMED VIDEO DISPLAY SYSTEM
`
`This application claims priority under 35 U.S.C 119 (e)
`of a provisional application entitled VCR LIKE FUNC
`TIONS FOR RENDERING VIDEO ON DEMAND (VOD)
`?led Jan. 30, 1997 by inventors Anthony W. Cannon, Anders
`E. Klemets, Hemanth S. Ravi, and David del Val
`(Application No. 60/036,661) and a provisional application
`entitled “METHODS AND APPARATUS FOR AUTODE
`TECTING PROTOCOLS IN A COMPUTER NETWORK”
`(Our Ref. No. VXTMP002+) ?led Jan. 30, 1997 by inven
`tors Anthony W. Cannon, Anders E. Klemets, Hemanth S.
`Ravi, and David del Val (Application No. 60/036,662).
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is related to co-pending US. patent
`application Ser. No. 08/818,805, entitled “Method and
`Apparatus for Implementing Motion Detection in Video
`Compression,”U.S. patent application Ser. No. 08/819,507,
`entitled “Digital Video Signal Encoder and Encoding
`Method,” US. patent application Ser. No. 08/818,804,
`entitled “Production of a Video Stream with Synchronized
`Annotations over a Computer Network,” US. patent appli
`cation Ser. No. 08/818,769, entitled “Methods and Appara
`tus for Automatically Detecting Protocols in a Computer
`Network,” US. patent application Ser. No. 08/818,127,
`entitled “Dynamic Bandwidth Selection for Ef?cient Trans
`mission of Multimedia Streams in a Computer Network,”
`US. patent application Ser. No. 08/819,585, entitled
`“Streaming and Displaying of a Video Stream with Syn
`chroniZed Annotations over a Computer Network,” US.
`patent application Ser. No. 08/818,644, allowed on Jan. 20,
`1999, entitled “Selective Retransmission for Efficient and
`Reliable Streaming of Multimedia Packets in a Computer
`Network,” US. patent application Ser. No. 08/819,579,
`entitled “Method and Apparatus for Table-Based Compres
`sion with Embedded Coding,” US. patent application Ser.
`No. 08/819,587, entitled “Method and Apparatus for Imple
`menting Motion Estimation in Video Compression,” all ?led
`on Mar. 14, 1997, US. patent application Ser. No. 08/822,
`156, entitled “Method and Apparatus for Communication
`Media Commands and Data Using the HTTP Protocol,” ?led
`on Mar. 17, 1997, US. patent application Ser. No. 08/818,
`826, ?led on Mar. 14, 1997 and allowed on Nov. 23, 1998,
`entitled “Conditional Replenishment Mechanism for Digital
`Video Signal Encoding,” US. patent application Ser. No.
`08/623,299, ?led Mar. 28, 1996, US. patent application Ser.
`No. 08/625,650, ?led Mar. 29, 1996, and US. patent appli
`cation Ser. No. 08/714,447, ?led Sep. 16, 1996, which are all
`incorporated herein by reference in their entirety for all
`purposes.
`
`BACKGROUND OF THE INVENTION
`
`The present invention relates to improved techniques for
`displaying video images transmitted over a computer net
`work. More particularly, the present invention relates to
`improved methods and apparatus for implementing control
`features, such as play, rewind, fast forward, pause, stop,
`record, and the like, on a real-time video stream and/or live
`video stream delivered via a computer network from server
`computers to client computers.
`It is well known that digital video data may be manipu
`lated and rendered using computers. In a computer network,
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`6,014,706
`
`2
`e.g., a client-server computer network, one or more com
`puters may be employed to receive the analog video data
`(e.g., from a video camera), encode that analog video data to
`a suitable digital format, and store the digital video data ?le.
`Using a computer coupled to the network, a user may, at a
`subsequent time, request the pre-stored digital video data ?le
`for display on a video display associated with the client
`computer.
`As computers become more pervasive in the workplace
`and in the home, the demand for digital video services
`correspondingly increases. By way of example, it has been
`recogniZed that it is possible to employ networked comput
`ers as a mass communication medium whereby a pre-stored
`digital video ?le may be transmitted from a server to one or
`more client computers to permit the client computers to
`display the images after the ?le is received. This technology
`may be employed to, for example, deliver movie or training
`video clips from a central server to one or more client
`computers for display.
`In the above example, it is typically necessary for the
`client computer to receive the entire pre-stored digital video
`?le prior to rendering the images. Real-time video
`streaming, on the other hand, refers to the situation wherein
`the client computer renders the images while they are
`streamed from the server computer. In some applications,
`real-time video streaming is favored since it permits the user
`to begin viewing video frames shortly after requesting the
`video ?le instead of having to wait until the entire pre-stored
`?le is downloaded from the server computer.
`It has been found, however, that real-time video streaming
`is dif?cult to implement on heterogeneous, lossy networks
`such as corporate intranets or the Internet, i.e., the well
`known international computer network that links, among
`others, various military, governmental, educational,
`nonpro?t, industrial and ?nancial institutions, commercial
`enterprises, and individuals. This is because real-time digital
`video applications, as are all digital video applications, are
`resource-intensive to implement. Even with compression,
`the transmission of quality video clips (i.e., those with
`acceptable frame rate and frame quality) places a heavy
`bandwidth burden on the computer network. For that reason,
`real-time video streaming has traditionally been imple
`mented on proprietary and expensive networks that are
`capable of supporting a high bit rate (e.g., private high-speed
`local area networks (LAN) or dedicated data links).
`Furthermore, real-time video data is time-sensitive, i.e.,
`the data packets containing the real-time video data must be
`received in a timely manner in the proper sequence for
`acceptable display. In bandwidth limited networks, e.g.,
`corporate intranets which support a high number of users or
`heterogeneous, lossy public networks such as the aforemen
`tioned Internet, the time-sensitive nature of real-time digital
`video data poses special challenges. There is, for example,
`less time to retransmit a lost data packet because if the time
`for displaying a given data packet at the client computer has
`passed, there is little use for that data packet if and when it
`arrives.
`It has also been found that real-time digital video stream
`ing poses complex frame synchroniZation issues. Since the
`video frames to be displayed are not stored with the client
`computer, there is no pre-stored ?le on which to perform
`control features such as rewind, fast forward, play, and
`pause. Typically, the video frames necessary for performing
`these functions are requested from the server computer
`itself. Responsive to the control commands, the video
`frames necessary for performing the requested control fea
`
`PAGE 10 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`3
`ture are then streamed to the client computer, typically over
`the same data connection to minimize the latency associated
`With opening another data connection. As can be appreciated
`by those skilled, complications can arise While sWitching
`among different groups of video frames, some of Which may
`be Waiting to be sent at the server, Waiting to be displayed
`at the client, in transit through the netWork, or lost.
`The complexity involved in implementing control fea
`tures on real-time video stream is further compounded by
`the requirement of loW latency, Which is imposed by real
`time video applications, i.e., the requirement that any delay
`betWeen the time a given video frame is transmitted from the
`server computer and the moment it is rendered at the client
`computer be minimiZed. Unless these control features are
`properly implemented, undue latency may occur and/or the
`quality of the vieWing experience may degrade.
`All the challenges discussed above also apply to live
`video streaming. In live video streaming, the video data may
`be digitiZed in one location, then encoded and transmitted to
`a client computer (via a server) substantially instantaneously
`(With some delay due to the encoding and transmission of
`video frames) for display. Live video and real-time video
`may be thought of as subsets of streamed video since both
`live video frames and real-time video frames are rendered as
`they are streamed from the server. Live video rendering,
`hoWever, results in the display of most recently encoded
`video frames Whereas real-time video rendering may result
`in displaying past video frames of a currently recorded event
`or even an event that happened and Was recorded a long time
`ago. As can be appreciated, live video streaming applica
`tions are even more sensitive With respect to the data packets
`transmitted via the netWork. This is because the live event
`being record continues to unfold, and video frames related
`thereto continue to be formed and require to be displayed as
`time goes by.
`Frame synchroniZation issues pertaining to live video
`streaming are further complicated by the fact that the digital
`video data ?le at the server is being formed at the same time
`the video frames pertaining thereto are displayed at the
`client computer. This is because copies of video frames sent
`to the client computer are also stored in the server in a digital
`video data ?le for subsequent use. Accordingly, there are
`complexities involved When, for example, the user Wishes to
`sWitch from a live video vieWing mode to a reWind mode,
`vieW past video frames for a feW seconds, and fast forWard
`to the end of the still groWing digital video data ?le to again
`play live video frames. Because of the complexities
`involved, as Well as the bandWidth and latency requirements,
`prior art attempts at implementing control features on live
`video streams have largely been unsatisfactory. While this is
`true for most networks, it is particularly true for the Internet
`Wherein the transport netWork is typically lossy and outside
`the control of the oWner of the server and/or the client
`computer, and Wherein the bandWidth available is both
`limited and subject to ?uctuations.
`In vieW of the foregoing, there are desired improved
`methods and apparatus for implementing control features on
`real-time video streams and/or live video streams transmit
`ted via a computer netWork from server computer(s) to client
`computer(s).
`
`15
`
`25
`
`35
`
`45
`
`55
`
`SUMMARY OF THE INVENTION
`
`The invention relates, in one embodiment, to a method for
`transmitting streamed digital video data from a server. The
`server is con?gured for coupling to a client computer via a
`computer netWork. The method includes inputting a ?rst
`
`65
`
`6,014,706
`
`4
`plurality of data packets into a server play-out buffer of the
`server. The ?rst plurality of the data packets contains video
`frames representing the streamed digital video data. An
`output of the server play-out buffer is con?gured to be
`coupled to a netWork data connection for transmitting the
`?rst plurality of the data packets to the client computer.
`The method includes receiving, using a retransmission
`buffer, the ?rst plurality of the data packets from the server
`play-out buffer. An output of the retransmission buffer is
`coupled to the netWork data connection. The method further
`includes outputting the ?rst plurality of the data packets
`from the server play-out buffer onto the netWork data
`connection for transmitting the data packets to the client
`computer via the computer netWork.
`In another embodiment, the invention relates to a method
`for displaying streamed digital video data on a client com
`puter. The client computer is con?gured to receive the
`streamed digital video data from a server computer via a
`computer netWork. The streamed digital video data is trans
`mitted from the server computer to the client computer as a
`stream of video frames. The method includes receiving a
`?rst plurality of video frames at the client computer. The
`plurality of video frames represents a subset of the stream of
`video frames. The stream of video frames comprises inde
`pendent playable video frames and dependent playable
`video frames.
`The method further includes displaying the ?rst plurality
`of video frames on a video display terminal associated With
`the client computer. There is further included issuing a
`reWind command from the client computer to the server. The
`reWind command causes a second plurality of video frames
`of the stream of video frames different from the ?rst plurality
`of video frames to be streamed from the server computer to
`the client computer. The second plurality of video frames
`has been streamed at least once to the client computer.
`In yet another embodiment, the invention relates to a
`computer readable medium containing computer-readable
`instructions for implementing control features con?gured
`for controlling a display of streamed digital video data at a
`client computer as the client computer transitions from a ?rst
`control mode to a second control mode. The client computer
`is con?gured for coupling to a server computer via a
`computer netWork. The streamed digital video data is trans
`mitted from the server computer to the client computer as a
`stream of video frames comprising independent video
`frames and dependent video frames. The stream of video
`frames is encapsulated in a plurality of data packets each
`having a unique packet sequence number and a unique
`timestamp. The computer readable instructions include com
`puter readable instructions for sending a control command
`and a time parameter from the client computer to the server
`computer. The control command represents a command to
`the server to transmit a ?rst plurality of video frames of the
`stream of video frames to the client computer in accordance
`With the second control mode.
`The computer readable instructions also include computer
`readable instructions for ascertaining, responsive to the
`control command and using the server computer, a ?rst
`independent video frame to transmit to the client computer.
`The ?rst independent video frame is selected responsive to
`the time parameter. The computer readable instructions
`further include computer readable instructions for transmit
`ting from the server computer to the client computer a packet
`sequence number associated With the ?rst independent video
`frame. The computer readable instructions further include
`computer readable instructions for streaming the ?rst plu
`
`PAGE 11 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`6,014,706
`
`5
`rality of video frames of the stream of video frames starting
`from the ?rst independent video frame from the server
`computer to the client computer to permit the ?rst plurality
`of video frames to be displayed at the client computer.
`These and other features of the present invention Will be
`described in more detail beloW in the detailed description of
`the invention and in conjunction With the folloWing ?gures.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1A depicts in accordance With one embodiment of
`the present invention a computer netWork 1 suitable for
`implementing the inventive streamed video display tech
`nique.
`FIG. 1B illustrates an embodiment of the invention
`Wherein an encoder furnishes video data from a video source
`to multiple servers.
`FIG. 2 is a block diagram of an exemplar digital
`computer, representing a computer suitable for implement
`ing either the server computer or the client computer of the
`present invention.
`FIG. 3 illustrates, in accordance With one embodiment of
`the invention, a VXF-formatted ?le, representing a ?le
`suitable for streaming encoded video data from the source,
`e.g., a video camera, to the server and the client application.
`FIG. 4 depicts, in accordance With one embodiment of the
`invention, tWo video streams: a play stream and a fast
`forWard stream to facilitate discussion.
`FIG. 5 illustrates, in accordance With one embodiment of
`the invention, a simpli?ed ?oWchart illustrating the imple
`mentation of certain control features such as play and fast
`forWard.
`FIG. 6 illustrates the steps involved, in one embodiment
`of the present invention, to implement the step for ascer
`taining the ?rst data packet for sending for the real-time play
`mode.
`FIG. 7 illustrates the steps involved in implementing the
`step for ascertaining the ?rst data packet for sending for the
`live play mode, in accordance With one embodiment of the
`present invention.
`FIG. 8 illustrates the steps involved, in one embodiment
`of the present invention, to implement the step for ascer
`taining the ?rst data packet for sending When transitioning
`from the real-time play mode (or other modes except live
`play) to the fast forWard mode.
`FIG. 9 illustrates the steps involved, in one embodiment
`of the present invention, to implement the step for ascer
`taining the ?rst data packet for sending When transitioning
`from the fast forWard mode to the real-time play mode.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`
`The present invention Will noW be described in detail With
`reference to a feW preferred embodiments thereof as illus
`trated in the accompanying draWings. In the folloWing
`description, numerous speci?c details are set forth in order
`to provide a thorough understanding of the present inven
`tion. It Will be apparent, hoWever, to one skilled in the art,
`that the present invention may be practiced Without some or
`all of these speci?c details. In other instances, Well knoWn
`process steps have not been described in detail in order to
`not unnecessarily obscure the present invention.
`In accordance With one aspect of the present invention,
`there are provided improved techniques for streaming real
`time video data from the server computer to the client
`
`10
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`6
`computer for display. In accordance With this aspect of the
`present invention, digital data pertaining to the real-time
`video stream is transmitted from the server computer to the
`client computer as data packets in either one of tWo streams:
`a play stream and a fast forWard stream. As Will be explored
`in great details herein, the use of a separate fast forWard
`stream of video frames advantageously permits the fast
`forWard feature to be implemented With loWer bandWidth
`requirements and improved display quality. As the user
`sWitches from the play mode to the fast forWard mode and
`vice versa, data packets containing video data in either the
`play stream or the fast forWard stream are transmitted from
`the server computer to the client computer for display.
`The play stream includes frames Which, When played at
`the designated frame rate by the renderer application in the
`client computer (about 10 frames per second in one
`example), maximiZes quality While minimiZing the band
`Width requirement. This normal play mode represents the
`mode that the user normally employs to vieW the video
`frames (e.g., Watching a movie or a video clip). It should be
`understood that normal play is typically accompanied by
`sound (typically furnished to the client via a data connection
`different from the one used to transmit video), and perhaps
`other applets. The fast forWard stream includes frames
`Which, When played at the designated frame rate by the
`renderer application in the client computer, provides the user
`With a fast forWard effect While advantageously keeping the
`display quality higher and the bit rate loWer than Would have
`been possible had the play stream been employed for fast
`forWarding. This aspect of the invention is discussed in
`detail later herein. By Way of example, the fast forWard
`stream may be played at 5 frames per second, Which displays
`frame at ?ve times the play speed.
`In accordance With another aspect of the present
`invention, the data packets traverse at least tWo buffers prior
`to arriving at the render application in the client computer
`for display: a retransmit buffer at the server computer and a
`client play-out buffer at the client computer. In one
`embodiment, a server play-out buffer is provided at the
`server as Well to facilitate ef?cient data packet transmission.
`The use of the client play-out buffer and/or the server
`play-out buffer advantageously maintain(s) small supply of
`data packets available at the client ready for display, thereby
`minimiZing impacts on the vieWing experience due to tem
`porary ?uctuations in the netWork’s available bandWidth and
`the temporary disruptions to the transmission of data packets
`through the computer netWork, e.g., due to temporary net
`Work congestion.
`In accordance With yet another aspect of the present
`invention, there are provided novel and ef?cient implemen
`tations of control features, such as play, reWind, fast forWard,
`pause, stop, record, and/or the like. In one embodiment, the
`control features are implemented to maximiZe the user’s
`familiarity With common video cassette recorder (VCR)
`control features. Using the control features of the present
`invention, the user is able to control, in a user-friendly and
`intuitive manner, the transmission and display of the video
`frames pertaining to a real-time video stream or,
`advantageously, even a live video stream at the client
`computer. This aspect of the invention is particularly advan
`tageous since the user may, using the inventive technique,
`?exibly control the display of streamed real-time video
`frames Without being unduly constrained by the real-time
`nature of the data or the inherent time-sensitive nature of the
`transmitted real-time data packets. In one embodiment, the
`transition betWeen the different control modes, e.g., from
`play to fast forWard, from reWind to play, and the like, are
`
`PAGE 12 of 21
`
`PETITIONERS' EXHIBIT 1009
`
`

`

`6,014,706
`
`7
`made ef?cient and synchronized, thereby maximizing the
`display quality of video frames.
`To further facilitate discussion of the foregoing, FIG. 1A
`depicts in accordance With one embodiment of the present
`invention a computer netWork 100, representing a computer
`netWork suitable for implementing the inventive real-time
`video display technique. Computer netWork 100 may, for
`example, represent a portion of the aforementioned Internet
`or corporate intranet.
`In FIG. 1A, computer netWork 100 includes a server
`computer 102 and a client computer 104. There is also
`shoWn a video camera 106. In the present example, video
`camera 106 represents the device for recording video data.
`The recorded video data may then be digitiZed and encoded
`by encoder 110 into the proper digital data format for
`transmission to either server 102 or memory 115 for storage.
`Encoder 110 represents, in one embodiment of the invention,
`the video source from Which data may be streamed to the
`client via the server. Encoder 110, Which may be imple
`mented in hardWare or softWare, may also perform com
`pression on the raW digital video data to improve storage and
`transmission ef?ciency. One suitable encoding scheme is
`disclosed in a commonly assigned co-pending patent US.
`patent application Ser. No. 08/623,299, ?led Mar. 28, 1996,
`incorporated herein by reference for all purposes.
`Data packets outputted by encoder 110 (or retrieved from
`memory 115) are then buffered Within server play-out buffer
`112 for transmission to client computer 104. Although
`memory 115 is depicted as nonvolatile disk memory in FIG.
`1A, it may represent any suitable type of memory device,
`including volatile semiconductor memory. As Will be dis
`cussed earlier, the ?le of data packets stored Within memory
`115 may be employed by client computer 104 to facilitate
`reWind, fast forWard, and other control modes.
`As each data packet or group of data packets is outputted
`from server play-out buffer 112 onto data connection 114 for
`transmission (e.g., responsive to a command from client
`computer 104 Which is received by server computer 102 via
`a control connection 116), the same data packet or group of
`data packets is input into retransmit buffer 118 at the server.
`Control connection 116 and data connection 114 have been
`discussed in detail in a commonly assigned co-pending US.
`patent application Ser. No. 08/819,586, ?led on Mar. 14,
`1997, entitled “Methods and Apparatus for Implementing
`Control Functions in a Streamed Video Display System.”
`Retransmit buffer 118 represents in one embodiment a
`?rst-in-?rst-out (FIFO) buffer Which retains for a limited
`time a data packet transmitted from server play-out buffer
`112. As neW data packets are input into retransmit buffer
`118, old data packets (starting With the oldest data packets)
`are discarded from transmit buffer 118. The use of the
`retransmit buffer advantageously facilitates the rapid
`retransmission of a data packet therein if that data packet is
`requested by client computer 104 for retransmission (e.g., in
`the event a data packet is detected to be missing by client
`computer 104). Retransmit buffer 118 is preferably siZed
`such that a data packet stays in retransmit buffer 118 slightly
`longer than the average latency period betWeen the time a
`data packet is transmitted from server 102 and When it is
`displayed at client computer 104. There is no need, in one
`embodiment of the invention, for the retransmit buffer 118
`to be much larger than mentioned since, due to the time
`sensitive nature of real-time video and/or live video, it is
`typically not useful to keep a data packet therein long past
`the time it is required at client computer 104 for display.
`As data packets are received by client computer 104 from
`data connection 114, they are inputted into client play-out
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`8
`buffer 120 to be displayed by renderer application 122.
`Client pl