Ulllted States Patent
`
`[19]
`
`[11] Patent Number:
`
`6,014,706
`
`Cannon et al.
`
`[45] Date of Patent:
`
`Jan. 11, 2000
`
`US006014706A
`
`[75]
`
`[54] METHODS AND APPARATUS FOR
`IMPLEMENTING CONTROL FUNCTIONS IN
`A STREAMED VIDEO DISPLAY SYSTEM
`,
`_
`IHVGIHOFSI AI1th0I1y C3I1I10I1; D3Vld del V31, b01h
`of Mountain View; Anders Klemets,
`Sunnyvale’ an of Calif.
`_
`_
`,
`[73] Assignee: Microsoft Corporation, Redmond,
`Wash.
`
`...........................
`V4/1997 DuLac el al.
`5,625,405
`~-
`I]3I1%he ettal-1
`..
`..
`e son e a.
`,
`,
`.. 348/426
`.
`4/1998 Kandlur et al.
`5,742,347
`6/1998 Ran ................... ..
`.. 709/247
`5,768,533
`5,828,848 10/1998 MacCormack et al.
`.............. .. 709/247
`
`
`
`348/47
`
`FOREIGN PATENT DOCUMENTS
`0605115
`7/1994 European Pat. Off.
`.
`0653884
`5/1995
`European pot. Off.
`.
`0676898 10/1995
`European Pat. Off.
`.
`0746158 12/1996
`European Pat. Off.
`.
`
`OTHER PUBLICATIONS
`
`[21] Appl. No: 03/819,586
`
`[22]
`
`1:115:93
`
`M31‘ 14: 1997
`
`[60]
`
`[51]
`
`Related U-S- Applicatiml Data
`Provisional application No. 60/036,662, Jan. 30, 1997, and
`provisional application No. 60/036,661, Jan. 30, 1997.
`Int Cl-7 ------------------------------------------------------ G061: 13/00
`US: Cl.
`........................ ..
`
`709/248
`[58] Field of Search ....................... .. 395/200.48, 200.61,
`395/200.63, 200.59, 200.65, 182.16; 709/217,
`218, 219, 229, 231, 233, 235, 248; 371/30,
`32> 33; 707/10
`
`,
`[56]
`
`_
`References Clted
`Us. PATENT DOCUMENTS
`
`Chen, H.J., et al., “A Scalable Video—011—Dei1iand Service
`for the Provision of VCR—Like Functions”, IEEE Proceed-
`ings of the International Conference on Multimedia Com-
`puting and S_}vs[ems7 Washington, DC, 65-72, (May 15-18,
`1995).
`Primary Examiner—Frank J. Asta
`Assistant ExaminerTJaS0n
`Cardong
`Attorney, Agent,
`or
`Firm—Schewgman,
`Woesnner & Kluth, P.A.
`_
`ABSTRACT
`[37]
`A method for displaying streamed digital video data on a
`client computer. The client computer is configured to receive
`the streamed digital video data from a server computer via
`a computer network. The streamed digital video data is
`transmitted from the server computer to the client computer
`as a stream of video frames. The method includes receiving
`a first plurality of video frames at the client computer. The
`plurality of video frames represents a subset of the stream of
`video frames. The stream of video frames comprises inde-
`pendent playable video frames and dependent playable
`/
`.d
`f
`h
`h d fu h
`.
`1 d
`d.
`1
`.
`h
`395/200
`vi eo rames. T e. met o
`rt er incu es.
`isp ayingt e
`348/7
`first plurality of video frames on a video display terminal
`N 370/17
`associated with the client computer. There is further
`395/650
`included issuing a rewind command from the client com-
`370/94.2
`puter to the server. The rewind command causes a second
`. 395/200.1
`plurality of video frames of the stream of video frames
`370/604
`different from the first plurality of video frames to be
`370/604
`Z Streamed from the server computer to the client computer.
`N 370/7'9
`The second plurality of video frames has been streamed at
`.. 370/84
`19315‘ 011“ to the C116“ °°mP“““~r~
`370/68.1
`........................ .. 395/806
`
`Liindberg,
`
`28 Claims, 8 Drawing Sheets
`
`.............................. .. 364/513
`6/1990 Isle et al.
`4,931,950
`---- --
`-- 370/50
`9/4991 G0]?/Stflnl
`5,050,151
`641992 B9991 9‘ 31-
`--
`395/154
`591199474
`gemjl ,et a1'1 “
`5374758
`’
`“Sum et a ‘
`5311454
`8/1994 Gelman et al.
`5,341,474
`51,4995 Hooper et al.
`5414 455
`7/1995 Chnnonto, Jr" et al.
`5,434,848
`5:455:910 10/1995 Johnson ot o1:
`5,481,542
`1/1996 Logston et al.
`5,490,252
`2/1996 Macera et al.
`5,504,744
`4/1996 Adams et a1~
`5519701
`5':1996 C°1m‘mt'°t a1‘ "
`ghcndett ‘:11.
`7’,1996 B33553: eteafi
`10/1996 Davis .......... ..
`11/1996 Bales et al.
`4/1997 Palmer et al.
`
`
`
`"
`
`5’537’408
`5:566:175
`5,574,724
`5,623,690
`
`_
`
`ENOODER
`
`I06
`
`WE)
`100
`
`/
`
`I02
`
`
`
`12345«
`148 RETRANSMISSION
`aurrzn
`
`
`SERVER
`
`\
`
`\
`CLIENT P\.AV—0UT
`BUFFER
`
`
`J
`/
`
`A
`
`RPX Exhibit 1042
`RPX Exhibit 1042
`RPX v. DAE
`RPX V. DAE
`
`

`
`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 1 of8
`
`6,014,706
`
`110
`
`
`
`ENCODER
`
`102
`
`SERVER
`
`1 18 RETRAN
`
`SMISSION
`
`
`
`
`
`
`BU
`
`FFER
`
`CLIENT
`
`122
`
`RENDERER
`
`FIG. 1A
`
`

`
`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 2 of 8
`
`6,014,706
`
`
`
`m:.9...
`
`+9
`
`

`
`P&U
`
`Sheet 3 of 8
`
`6,014,706
`
`
`
`I
`
`O N
`
`tH..&NmaHn_.............................................-LwAa8m
`
`._
`
`
`»mw.vm_2HH__
`_«mmH__0_wonanMH.<~_w__m__~_m_._mommmooEomo_z
`8NI_EHJ__
`
`_
`
`__H__H
`
`
`

`
`aP3U
`
`6
`
`M7,4m,
`
`Mcom
`
`
`
`
`
`F/\|\/[I/\.|I.\
`
`
`
`(I/\|I\mEconSymonA38»EnNon
`
`
`Mo.nmomo_nmomW...V$95:$95:
`
`5Rzo_mm_mn_:oon._.__.._
`
`M.o_.._
`
`
`

`
`tHEtaP5U
`
`Jan. 11, 2000
`
`Sheet 5 of 8
`
`607,410’6
`
`
`
`zggg@@@m@@@2
`
`
`
`¢N¢um.‘ON.‘m30?3.‘NT‘03.mo...mo...3...No.»//00¢
`
`¢.9...
`
`sagggEgg2
`m3m9.1:.N38.»mm;mm?fiemm.‘\\on¢
`
`
`

`
`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 6 of 8
`
`6,014,706
`
`® 500
`
`CLIENT SENDS COMMAND
`AND TIME PARAMETER
`
`502
`
`SERVER ASCERTAINS FIRST
`DATA PACKET TO SEND
`
`504
`
`SERVER AND CLIENT
`FLUSH BUFFERS
`
`SERVER COMMUNICATES TO
`CLIENT FIRST PACKET
`SEQUENCE NUMBER
`
`505
`
`508
`
`SERVER STREAMS PACKETS
`To CLIENT
`
`510
`
`CLIENT DISPLAYS
`STREAMED DATA PACKETS
`
`512
`
`@ 514
`
`FIG. 5
`
`

`
`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 7 of 8
`
`6,014,706
`
`530
`
`504
`
`
`
`
`SERVER SEEKS BACKWARD/FORWARD
`FOR SEEKABLE FRAME lMMEDlATELY
`NEXT TO FRAME ASSOCIATED
`WITH TIME PARAMETER OF
`STEP 502
`
`
` 532
`
`
`
`
`
`
`DESIGNATE SEEKABLE FRAME
`OF STEP 532 FIRST FRAME
`TO SEND TO CLIENT
`
` 534
`
`
`
` SERVER WAITS FOR NEXT
`
`SEEKABLE FRAME FROM
`ENCODER
`
`
`
`
`
` DESIGNATE SEEKABLE FRAME
`
` 554
`OF STEP 552 FIRST FRAME
`TO SEND TO CLIENT
`
`552
`
`
`
` FIG. 7
`
`

`
`U.S. Patent
`
`Jan. 11,2000
`
`Sheet 8 of 8
`
`6,014,706
`
`800
`
`
`
`
`SERVER SEEKS BACK IN FAST FORWARD
`STREAM FOR FIRST SEEKABLE FRAME
`STARTING FROM FRAME WHOSE TIME
`STAMP MOST CLOSELY CORRESPONDS TO
`TIME STAMP OF STEP 502
`
`
`
`
`
`802
`
`804
`
`
`
`
`
`
`
`DESIGNATE SEEKABLE FRAME
`OF STEP 802 FIRST FRAME
`TO SEND TO CLIENT
`
`
`
`902
`
`
`
`
`
`
`SERVER SEEKS BACK IN PLAY
`STREAM FOR FIRST SEEKABLE FRAME
`
`
`STARTING FROM FRAME WHOSE TIME
`STAMP MOST CLOSELY CORRESPONDS TO
`TIME STAMP OF STEP 502
`
` DESIGNATE SEEKABLE FRAME
`
` 904
`OF STEP 902 FIRST FRAME
`TO SEND TO CLIENT
`
` FIG. 9
`
`

`
`6,014,706
`
`1
`METHODS AND APPARATUS FOR
`IMPLEMENTING CONTROL FUNCTIONS IN
`A STREAMED VIDEO DISPLAY SYSTEM
`
`This application claims priority under 35 U.S.C 119 (e)
`of a provisional application entitled VCR LIKE FUNC-
`TIONS FOR RENDERING VIDEO ON DEMAND (VOD)
`filed Jan. 30, 1997 by inventors Anthony W. Cannon, Anders
`E. Klemets, Hemanth S. Ravi, and David del Val
`(Application No. 60/036,661) and a provisional application
`entitled “METHODS AND APPARATUS FOR AUTODE-
`TECTING PROTOCOLS IN A COMPUTER NETWORK”
`
`(Our Ref. No. VXTMP002+) filed Jan. 30, 1997 by inven-
`tors Anthony W. Cannon, Anders E. Klemets, Hemanth S.
`Ravi, and David del Val (Application No. 60/036,662).
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is related to co-pending U.S. patent
`application Ser. No. 08/818,805, entitled “Method and
`Apparatus for Implementing Motion Detection in Video
`Compression,”U.S. patent application Ser. No. 08/819,507,
`entitled “Digital Video Signal Encoder and Encoding
`Method,” U.S. patent application Ser. No. 08/818,804,
`entitled “Production of a Video Stream with Synchronized
`Annotations over a Computer Network,” U.S. patent appli-
`cation Ser. No. 08/818,769, entitled “Methods and Appara-
`tus for Automatically Detecting Protocols in a Computer
`Network,” U.S. patent application Ser. No. 08/818,127,
`entitled “Dynamic Bandwidth Selection for Efficient Trans-
`mission of Multimedia Streams in a Computer Network,”
`U.S. patent application Ser. No. 08/819,585, entitled
`“Streaming and Displaying of a Video Stream with Syn-
`chronized Annotations over a Computer Network,” U.S.
`patent application Ser. No. 08/818,644, allowed on Jan. 20,
`1999, entitled “Selective Retransmission for Efficient and
`Reliable Streaming of Multimedia Packets in a Computer
`Network,” U.S. patent application Ser. No. 08/819,579,
`entitled “Method and Apparatus for Table-Based Compres-
`sion with Embedded Coding,” U.S. patent application Ser.
`No. 08/819,587, entitled “Method and Apparatus for Imple-
`menting Motion Estimation in Video Compression,” all filed
`on Mar. 14, 1997, U.S. patent application Ser. No. 08/822,
`156, entitled “Method and Apparatus for Communication
`Media Commands and Data Using the HTTP Protocol,” filed
`on Mar. 17, 1997, U.S. patent application Ser. No. 08/818,
`826, filed on Mar. 14, 1997 and allowed on Nov. 23, 1998,
`entitled “Conditional Replenishment Mechanism for Digital
`Video Signal Encoding,” U.S. patent application Ser. No.
`08/623,299, filed Mar. 28, 1996, U.S. patent application Ser.
`No. 08/625,650, filed Mar. 29, 1996, and U.S. patent appli-
`cation Ser. No. 08/714,447, filed Sep. 16, 1996, which are all
`incorporated herein by reference in their entirety for all
`purposes.
`
`BACKGROUND OF THE INVENTION
`
`The present invention rclatcs to improved techniques for
`displaying video images transmitted over a computer net-
`work. More particularly,
`the present invention relates to
`improved methods and apparatus for implementing control
`features, such as play, rewind, fast forward, pause, stop,
`record, and the like, on a real-time video stream and/or live
`video stream delivered via a computer network from server
`computers to client computers.
`It is well known that digital video data may be manipu-
`lated and rendered using computers. In a computer network,
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`e.g., a client-server computer network, one or more coni-
`puters may be employed to receive the analog video data
`(e.g., from a video camera), encode that analog video data to
`a suitable digital format, and store the digital video data file.
`Using a computer coupled to the network, a user may, at a
`subsequent time, request the pre-stored digital video data file
`for display on a video display associated with the clicnt
`computer.
`As computers become more pervasive in the workplace
`and in the home,
`the demand for digital video services
`correspondingly increases. By way of example, it has been
`recognized that it is possible to employ networked comput-
`ers as a mass communication medium whereby a pre-stored
`digital video file may be transmitted from a server to one or
`more client computers to permit the client computers to
`display the images after the file is received. This technology
`may be employed to, for example, deliver movie or training
`video clips from a central server to one or more client
`computers for display.
`In the above example, it is typically necessary for the
`client computer to receive the entire prc-storcd digital video
`file prior to rendering the images. Real-time video
`streaming, on the other hand, refers to the situation wherein
`the client computer renders the images while they are
`streamed from the server computer. In some applications,
`real-time video streaming is favored since it permits the user
`to begin viewing video frames shortly after requesting the
`video file instead of having to wait until the entire pre-stored
`file is downloaded from the server computer.
`It has been found, however, that real-time video streaming
`is diflicult to implement on heterogeneous, lossy networks
`such as corporate intranets or the Internet, i.e., the well-
`known international computer network that links, among
`others, various military, governmental, educational,
`nonprofit, industrial and financial institutions, commercial
`enterprises, and individuals. This is because real-time digital
`video applications, as are all digital video applications, are
`resource-intensive to implement. Even with compression,
`the transmission of quality video clips (i.e.,
`those with
`acceptable frame rate and frame quality) places a heavy
`bandwidth burden on the computer network. For that reason,
`real-time video streaming has traditionally been imple-
`mented on proprietary and expensive networks that are
`capable of supporting a high bit rate (e.g., private high-speed
`local area networks (LAN) or dedicated data links).
`Furthermore, real-time video data is time-sensitive, i.e.,
`the data packets containing the real-time video data must be
`received in a timely manner in the proper sequence for
`acceptable display. In bandwidth limited networks, e.g.,
`corporate intranets which support a high number of users or
`heterogeneous, lossy public networks such as the aforemen-
`tioned Internet, the time-sensitive nature of real-time digital
`video data poses special challenges. There is, for example,
`less time to rctransmit a lost data packet because if the time
`for displaying a given data packet at the client computer has
`passed, there is little use for that data packet if and when it
`arrives.
`
`It has also been found that real-time digital video stream-
`ing poses complex frame synchronization issues. Since the
`video frames to be displayed are not stored with the client
`computer, there is no pre-stored file on which to perform
`control features such as rewind, fast forward, play, and
`pause. Typically, the video frames necessary for performing
`these functions are requested from the server computer
`itself. Responsive to the control commands,
`the video
`frames necessary for performing the requested control fea-
`
`

`
`6,014,706
`
`3
`ture are then streamed to the client computer, typically over
`the same data connection to minimize the latency associated
`with opening another data connection. As can be appreciated
`by those skilled, complications can arise while switching
`among different groups of video frames, some of which may
`be waiting to be sent at the server, waiting to be displayed
`at the client, in transit through the network, or lost.
`The complexity involved in implementing control fea-
`tures on real-time video stream is further compounded by
`the requirement of low latency, which is imposed by real-
`time video applications, i.e., the requirement that any delay
`between the time a given video frame is transmitted from the
`server computer and the moment it is rendered at the client
`computer be minimized. Unless these control features are
`properly implemented, undue latency may occur and/or the
`quality of the viewing experience may degrade.
`All
`the challenges discussed above also apply to live
`video streaming. In live video streaming, the video data may
`be digitized in one location, then encoded and transmitted to
`a client computer (via a server) substantially instantaneously
`(with some delay due to the encoding and transmission of
`video frames) for display. Live video and real-time video
`may be thought of as subsets of streamed video since both
`live video frames and real-time video frames are rendered as
`
`they are streamed from the server. Live video rendering,
`however, results in the display of most recently encoded
`video frames whereas real-time video rendering may result
`in displaying past video frames of a currently recorded event
`or even an event that happened and was recorded a long time
`ago. As can be appreciated, live video streaming applica-
`tions are even more sensitive with respect to the data packets
`transmitted via the network. This is because the live event
`being record continues to unfold, and video frames related
`thereto continue to be formed and require to be displayed as
`time goes by.
`Frame synchronization issues pertaining to live video
`streaming are further complicated by the fact that the digital
`video data file at the server is being formed at the same time
`the video frames pertaining thereto are displayed at
`the
`client computer. This is because copies of video frames sent
`to the client computer are also stored in the server in a digital
`video data file for subsequent use. Accordingly, there are
`complexities involved when, for example, the user wishes to
`switch from a live video viewing mode to a rewind mode,
`view past video frames for a few seconds, and fast forward
`to the end of the still growing digital video data file to again
`play live video frames. Because of the complexities
`involved, as well as the bandwidth and latency requirements,
`prior art attempts at implementing control features on live
`video streams have largely been unsatisfactory. While this is
`true for most networks, it is particularly true for the Internet
`wherein the transport network is typically lossy and outside
`the control of the owner of the server and/or the client
`computer, and wherein the bandwidth available is both
`limited and subject to fluctuations.
`In View of the foregoing,
`there are desired improved
`methods and apparatus for implementing control features on
`real-time video streams and/or live video streams transmit-
`ted via a computer network from server computer(s) to client
`computer(s).
`SUMMARY OF THE INVENTION
`
`The invention relates, ir1 one enibodinient, to a method for
`transmitting streamed digital video data from a server. The
`server is configured for coupling to a client computer via a
`computer network. The method includes inputting a first
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`plurality of data packets into a server play-out buffer of the
`server. The first plurality of the data packets contains video
`frames representing the streamed digital video data. An
`output of the server play-out buffer is configured to be
`coupled to a network data connection for transmitting the
`first plurality of the data packets to the client computer.
`The method includes receiving, using a retransmission
`buffer, the first plurality of the data packets from the server
`play-out buffer. An output of the retransmission buffer is
`coupled to the network data connection. The method further
`includes outputting the first plurality of the data packets
`from the server play-out buffer onto the network data
`connection for transmitting the data packets to the client
`computer via the computer network.
`In another embodiment, the invention relates to a method
`for displaying streamed digital video data on a client com-
`puter. The client computer is configured to receive the
`streamed digital video data from a server computer via a
`computer network. The streamed digital video data is trans-
`mitted from the server computer to the client computer as a
`stream of video frames. The method includes receiving a
`first plurality of video frames at the client computer. The
`plurality of video frames represents a subset of the stream of
`video frames. The stream of video frames comprises inde-
`pendent playable video frames and dependent playable
`video frames.
`
`The method further includes displaying the first plurality
`of video frames on a video display terminal associated with
`the client computer. There is further included issuing a
`rewind command from the client computer to the server. The
`rewind command causes a second plurality of video frames
`of the stream of video frames different from the first plurality
`of video frames to be streamed from the server computer to
`the client computer. The second plurality of video frames
`has been streamed at least once to the client computer.
`In yet another embodiment, the invention relates to a
`computer readable medium containing computer-readable
`instructions for implementing control features configured
`for controlling a display of streamed digital video data at a
`client computer as the client computer transitions from a first
`control mode to a second control mode. The client computer
`is configured for coupling to a server computer via a
`computer network. The streamed digital video data is trans-
`mitted from the server computer to the client computer as a
`stream of video frames comprising independent video
`frames and dependent video frames. The stream of video
`frames is encapsulated in a plurality of data packets each
`having a unique packet sequence number and a unique
`timestamp. The computer readable instructions include com-
`puter readable instructions for sending a control command
`and a time parameter from the client computer to the server
`computer. The control command represents a command to
`the server to transmit a first plurality of video frames of the
`stream of video frames to the client computer in accordance
`with the second control mode.
`
`The computer readable instructions also include computer
`readable instructions for ascertaining, responsive to the
`control command and using the server computer, a first
`independent video frame to transmit to the client computer.
`The first independent video frame is selected responsive to
`the time parameter. The computer readable instructions
`further include computer readable instructions for transmit-
`ting from the server computer to the client computer a packet
`sequence number associated with the first independent video
`frame. The computer readable instructions further include
`computer readable instructions for streaming the first plu-
`
`

`
`6,014,706
`
`5
`rality of video frames of the stream of video frames starting
`from the first independent video frame from the server
`computer to the client computer to permit the first plurality
`of video frames to be displayed at the client computer.
`These and other features of the present invention will be
`described in more detail below in the detailed description of
`the invention and in conjunction with the following figures.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1A depicts in accordance with one embodiment of
`the present
`invention a computer network 1 suitable for
`implementing the inventive streamed video display tech-
`nique.
`FIG. 1B illustrates an embodiment of the invention
`wherein an encoder furnishes video data from a video source
`
`to multiple servers.
`FIG. 2 is a block diagram of an exemplar digital
`computer, representing a computer suitable for implement-
`ing either the server computer or the client computer of the
`present invention.
`FIG. 3 illustrates, in accordance with one embodiment of
`the invention, a VXF-formatted file, representing a file
`suitable for streaming encoded video data from the source,
`e.g., a video camera, to the server and the client application.
`FIG. 4 depicts, in accordance with one embodiment of the
`invention,
`two video streams: a play stream and a fast-
`forward stream to facilitate discussion.
`
`FIG. 5 illustrates, in accordance with one embodiment of
`the invention, a simplified flowchart illustrating the imple-
`mentation of certain control features such as play and fast
`forward.
`
`FIG. 6 illustrates the steps involved, in one embodiment
`of the present invention, to implement the step for ascer-
`taining the first data packet for sending for the real-time play
`mode.
`
`FIG. 7 illustrates the steps involved in implementing the
`step for ascertaining the first data packet for sending for the
`live play mode, in accordance with one embodiment of the
`present invention.
`FIG. 8 illustrates the steps involved, in one embodiment
`of the present invention, to implement the step for ascer-
`taining the first data packet for sending when transitioning
`from the real-time play mode (or other modes except live
`play) to the fast forward mode.
`FIG. 9 illustrates the steps involved, in one embodiment
`of the present invention, to implement the step for ascer-
`taining the first data packet for sending when transitioning
`from the fast forward mode to the real-time play mode.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`
`The present invention will now be described in detail with
`reference to a few preferred embodiments thereof as illus-
`trated in the accompanying drawings.
`In the following
`description, numerous specific details are set forth in order
`to provide a thorough understanding of the present inven-
`tion. It will be apparent, however, to one skilled in the art,
`that the present invention may be practiced without some or
`all of these specific details. In other instances, well known
`process steps have not been described in detail in order to
`not unnecessarily obscure the present invention.
`In accordance with one aspect of the present invention,
`there are provided improved techniques for streaming real-
`time video data from the server computer to the client
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`computer for display. In accordance with this aspect of the
`present invention, digital data pertaining to the real-time
`video stream is transmitted from the server computer to the
`client computer as data packets in either one of two streams:
`a play stream and a fast forward stream. As will be explored
`in great details herein, the use of a separate fast forward
`stream of video frames advantageously permits the fast
`forward feature to be implemented with lower bandwidth
`requirements and improved display quality. As the user
`switches from the play mode to the fast forward mode and
`vice versa, data packets containing video data in either the
`play stream or the fast forward stream are transmitted from
`the server computer to the client computer for display.
`The play stream includes frames which, when played at
`the designated frame rate by the renderer application in the
`client computer (about 10 frames per second in one
`example), maximizes quality while minimizing the band-
`width requirement. This normal play mode represents the
`mode that the user normally employs to view the video
`frames (e.g., watching a movie or a video clip). It should be
`understood that normal play is typically accompanied by
`sound (typically furnished to the client via a data connection
`different from the one used to transmit video), and perhaps
`other applets. The fast forward stream includes frames
`which, when played at the designated frame rate by the
`renderer application in the client computer, provides the user
`with a fast forward effect while advantageously keeping the
`display quality higher and the bit rate lower than would have
`been possible had the play stream been employed for fast
`forwarding. This aspect of the invention is discussed in
`detail later herein. By way of example,
`the fast forward
`stream may be played at 5 frames per second, which displays
`frame at five times the play speed.
`In accordance with another aspect of the present
`invention, the data packets traverse at least two buffers prior
`to arriving at the render application in the client computer
`for display: a retransmit buffer at the server computer and a
`client play-out buffer at
`the client computer.
`In one
`embodiment, a server play-out buffer is provided at the
`server as well to facilitate efficient data packet transmission.
`The use of the client play-o11t buffer and/or the server
`play-out buffer advantageously maintain(s) small supply of
`data packets available at the client ready for display, thereby
`minimizing impacts on the viewing experience due to tem-
`porary fluctuations in the network’s available bandwidth and
`the temporary disruptions to the transmission of data packets
`through the computer network, e.g., due to temporary net-
`work congestion.
`In accordance with yet another aspect of the present
`invention, there are provided novel and efficient implemen-
`tations of control features, such as play, rewind, fast forward,
`pause, stop, record, and/or the like. In one embodiment, the
`control features are implemented to maximize the user’s
`familiarity with common video cassette recorder (VCR)
`control features. Using the control features of the present
`invention, the user is able to control, in a user-friendly and
`intuitive manner, the transmission and display of the video
`frames pertaining to a
`real-time video stream or,
`advantageously, even a live video stream at
`the client
`computer. This aspect of the invention is particularly advan-
`tageous since the user may, using the inventive technique,
`flexibly control
`the display of streamed real-time video
`frames without being unduly constrained by the real-time
`nature of the data or the inherent time-sensitive nature of the
`transmitted real-time data packets. In one embodiment, the
`transition between the different control modes, e.g., from
`play to fast forward, from rewind to play, and the like, are
`
`

`
`6,014,706
`
`7
`made efficient and synchronized, thereby maximizing the
`display quality of video frames.
`To further facilitate discussion of the foregoing, FIG. 1A
`depicts in accordance with one embodiment of the present
`invention a computer network 100, representing a computer
`network suitable for implementing the inventive real-time
`video display technique. Computer network 100 may, for
`example, represent a portion of the aforementioned Internet
`or corporate intranet.
`In FIG. 1A, computer network 100 includes a server
`computer 102 and a client computer 104. There is also
`shown a video camera 106. In the present example, video
`camera 106 represents the device for recording video data.
`The recorded video data may then be digitized and encoded
`by encoder 110 into the proper digital data format for
`transmission to either server 102 or memory 115 for storage.
`Encoder 110 represents, in one embodiment of the invention,
`the video source from which data may be streamed to the
`client via the server. Encoder 110, which may be imple-
`mented in hardware or software, may also perform com-
`pression on the raw digital video data to improve storage and
`transmission efliciency. One suitable encoding scheme is
`disclosed in a commonly assigned co—pending patent U.S.
`patent application Ser. No. 08/623,299, filed Mar. 28, 1996,
`incorporated herein by reference for all purposes.
`Data packets outputted by encoder 110 (or retrieved from
`memory 115) are then buffered within server play-out buffer
`112 for transmission to client computer 104. Although
`memory 115 is depicted as nonvolatile disk memory in FIG.
`1A, it may represent any suitable type of memory device,
`including volatile semiconductor memory. As will be dis-
`cussed earlier, the file of data packets stored within memory
`115 may be employed by client computer 104 to facilitate
`rewind, fast forward, and other control modes.
`As each data packet or group of data packets is outputted
`from server play-out buffer 112 onto data connection 114 for
`transmission (e.g., responsive to a command from client
`computer 104 which is received by server computer 102 via
`a control connection 116), the same data packet or group of
`data packets is input into retransmit buffer 118 at the server.
`Control connection 116 and data connection 114 have been
`discussed in detail in a commonly assigned co—pending U.S.
`patent application Ser. No. 08/819,586, filed on Mar. 14,
`1997, entitled “Methods and Apparatus for Implementing
`Control Functions in a Streamed Video Display System.”
`Retransmit buffer 118 represents in one embodiment a
`first-in-first-out (FIFO) buffer which retains for a limited
`time a data packet transmitted from server play-out buffer
`112. As new data packets are input into retransmit buffer
`118, old data packets (starting with the oldest data packets)
`are discarded from transmit buffer 118. The use of the
`
`retransmit buffer advantageously facilitates the rapid
`retransmission of a data packet therein if that data packet is
`requested by client computer 104 for retransmission (e.g., in
`the event a data packet is detected to be missing by client
`computer 104). Retransmit bufier 118 is preferably sized
`such that a data packet stays in retransmit buffer 118 slightly
`longer than the average latency period between the time a
`data packet is transmitted from server 102 and when it is
`displayed at client computer 104. There is no need, in one
`embodiment of the invention, for the retransmit buffer 118
`to be much larger than mentioned since, due to the time-
`sensitive nature of real-time video and/or live video, it is
`typically not usefiil to keep a data packet therein long past
`the time it is required at client computer 104 for display.
`As data packets are received by client computer 104 from
`data connection 114, they are inputted into client play-out
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`buffer 120 to be displayed by renderer application 122.
`Client play-out buffer 120 may represent,
`in one
`embodiment, a FIFO buffer. Client play-out buffer 120
`and/or server play-out buffer 112 are typically sized appro-
`priately to minimize latency while taking into account the
`reliability and stability of network 100 through which data
`connection 114 traverses. If the network is fairly reliable and
`its bandwidth is fairly stable, client play-out buffer 120
`and/or server play-out buffer 112 may be relatively small to
`minimize the latency between the time a data packet is
`outputted by encoder 110 and the time it is displayed at the
`client computer. On the other hand, a larger client play-out
`buffer 120 and/or server play-out buffer 112 may be able to
`more effectively insulate renderer application 122 from
`temporary bandwidth capacity fiuctuations and disruptions
`in the transmission of data packets from server computer
`102.
`
`Client play-out buffer 120 may be, but is not required to
`be, equal in size to retransmit buifer 118 since retransmis-
`sion of a data packet from retransmit buffer 118