(12) Unlted States Patent
`(10) Patent No.:
`US 7,237,254 B1
`
`Omoigui
`(45) Date of Patent:
`Jun. 26, 2007
`
`USOO7237254B1
`
`(54) SEAMLESS SWITCHING BETWEEN
`DIFFERENT PLAYBACK SPEEDS OF
`TIME-SCALE MODIFIED DATA STREAMS
`
`(75)
`
`Inventor: Nosakhare D. Omoigui, Redmond, WA
`(Us)
`
`(73) Assignee: Microsoft Corporation, Redmond, WA
`(US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 1 54(1)) by 0 days.
`
`(21) Appl. No.: 09/539,170
`.
`F11ed:
`
`(22)
`
`Mar. 29, 2000
`
`(51)
`
`Int- Cl-
`(200601)
`H04N 7/173
`(52) US. Cl.
`......................................... 725/94; 709/231
`(58) Field of Classification Search .................. 725/88,
`725/102: 93: 94
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`5,559,942 A
`5,566,175 A
`5,574,724 A
`5,606,359 A *
`
`9/1996 Gough et a1.
`............... 395/155
`10/1996 Davis ...........
`370/84
`
`................ 370/68.1
`11/1996 Bales et al.
`2/1997 Youden et al.
`................ 725/88
`
`5,614,940 A
`5,617,423 A
`5,623,690 A
`
`5,625,405 A
`5,640,320 A
`
`3/1997 Cobbley et a1.
`4/1997 Li et al.
`..........
`4/1997 Palmer et al.
`
`..... 348/7
`.. 370/426
`.. 395/806
`
`
`
`4/1997 DuLac et al.
`6/1997 Jackson et al.
`
`.................. 348/7
`............. 364/192
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`0605115
`7/1994
`0653884
`5/1995
`0 669 587 A2
`8/1995
`0676898
`10/1995
`0746158
`12/1996
`0 812 112 A2
`10/1997
`W0 94/01964
`1/1994
`wo 98/37698
`8/1998
`
`EP
`EP
`EP
`EP
`EP
`EP
`W0
`wo
`
`OTHER PUBLICATIONS
`
`H]. Chen et al., “A Scalable Video-on-Demand Service for the
`Provision ofVCR-Like Functions,” IEEE, May 15, 1995, pp. 65-72.
`
`U.S. PATENT DOCUMENTS
`
`(Continued)
`
`4,931,950 A
`5,050,161 A
`5,119,474 A
`5,274,758 A
`5,309,562 A
`5,313,454 A
`5,341,474 A
`5,4 14/55 A
`5,434,848 A
`5,455,910 A
`5’481’542 A
`5,490,252 A
`5 504 744 A
`5’519’701 A
`5,521,630 A
`5,533,021 A
`5,537,408 A
`5,541,955 A
`
`
`
`................... 364/513
`6/1990 Isle et al.
`370/60
`9/1991 Golestani
`6/1992 Beitel et al.
`.
`.. 395/154
`12/1993 Beitel et al.
`.
`395/154
`5/1994 Li
`..............
`395/200
`..
`5/1994 Bustini et al.
`..... 370/13
`-
`8/ 1994 Gelman et 31~
`-- 395/200
`5/1995 HO?!)er et al~ ~~~~~~~~~~~~~~~~~ 348/7
`7/1995 Chlmento, Jr' et 31'
`~~~~~~~ 370/17
`10/1995 Johnson et al.
`............. 395/650
`
`‘
`“1996 LOgSton et al'
`””” 370/94'2
`..
`2/1996 Macera et al.
`395/200.01
`
`4/ 1996 Ad
`t
`1
`370/60 1
`
`5/1996 Coliisn: 6:151 """"" 370/601
`5/1996 Chen et al.
`.................... 348/7
`
`7/1996 Branstad etal.
`..... 370/60.1
`7/1996 Branstad et al.
`.............. 370/79
`7/1996 Jacobsmeyer ............... 375/222
`
`Primary ExamineriAndrew Y Koenig
`.
`.
`i
`.
`(74) Ammey’ Agem’ 0r F’rmiperkms C016 LLP
`
`(57)
`
`ABSTRACT
`
`In a network environment, multimedia content is streamed
`from a server computer to a client computer via the network.
`Auser of the client computer can alter the speed at which the
`multimedia content is played; either speeding up 01' slowing
`down the playback. The client seamlessly switches between
`~
`~
`the d1fierent playback speeds, reduc1ng breaks and/or delays
`between the time the user selects the new playback speed
`Endkthe Eme the muglmedla content begms bemg played
`ac a” 6 new 513% ~
`
`16 Claims, 9 Drawing Sheets
`
`102
`
`
`
`-202Composite
`
`
`
`
`202
`Control
`Information
`
`104
`
`Media
`Stream
`
`
`
`Audio Render
`
`
`
`Video Render
`
`1
`
`NEULION 1022
`
`1
`
`NEULION 1022
`
`

`

`US 7,237,254 B1
`
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`
`
`9/1997 VIauldin et a1.
`5,664,227 A
`............ 395/778
`11/1997 Goldberg etal.
`5,692,213 A
`395/806
`
`5,699,474 A * 12/1997 Suzuki et a1. ........... 386/68
`
`5,717,691 A
`2/1998 Dighe etal~
`370/401
`5,717,869 A
`2/1998 Vloran etal ~~
`395/339
`....... 364/514
`5,719,786 A
`2/1998 \Ielson et a1.
`
`2/1998 Dunn etal ~~~~~~~
`~~ 395/20049
`5,721,829 A
`~~~~~~~ 348/426
`5742347 A
`4/1998 Kandluretal
`
`6/1998 Ran ..............
`5,768,533 A
`395/200.77
`7/1998 Vloran etal ~~~~~~~~~~~~ 345/328
`5786814 A
`
`5794110 A
`8/1998 Goldhaber et 31 ~-
`705/14
`~ 707/104
`5,794,249 A
`8/1998 Orsolini etal
`~~
`~~ 706/11
`5799292 A
`8/1998 Hekmatpour
`
`5,801,685 A
`9/1998 Vfilleretal
`345/302
`5,808,662 A
`9/1998 Kinney et a1.
`348/15
`..
`.......... 348/7
`5,818,510 A
`10/1998 Cobbley et a1.
`5,822,537 A
`10/1998 Katseff et a1.
`......... 395/200.61
`5,828,848 A
`10/1998 VIacCormack et al.
`395/200.77
`5,835,495 A
`11/1998 Ferriere ...................... 370/465
`......... 386/96
`5,835,667 A
`11/1998 Wactlar et a1.
`.
`
`5,838,906 A
`11/1998 Doyle et a1.
`395/200.32
`
`1/1999 Cave .................... 345/348
`5,859,641 A
`.. 395/200.77
`5,864,682 A
`1/1999 Porter et a1.
`5,870,755 A
`2/ 1999 Stevens et a1.
`....... 707/104
`2/1999 Yamada et a1.
`5,873,735 A
`434/316
`
`
`. 345/302
`5,892,506 A
`4/ 1999 Hermanson .
`5,903,673 A
`5/1999 Wang et a1.
`382/236
`
`6/1999 Klemets et a1.
`........ 395/182.16
`5,918,002 A
`7/ 1999 Minakuchi et a1.
`......... 707/4
`5,930,787 A
`5,953,506 A
`9/ 1999 Kalra et a1.
`............ 395/200.61
`.
`...... 707/10
`5,956,716 A
`9/1999 Kenner et a1.
`
`10/ 1999 Polish ...........
`.715/723
`5,963,202 A *
`
`..
`5,995,941 A
`11/1999 Maquire et a1.
`705/10
`......
`5,999,979 A
`12/ 1999 Vellanki et a1.
`709/232
`
`6,006,241 A
`12/1999 Purnaveja et a1.
`707/512
`6,014,706 A
`1/2000 Cannon et a1.
`709/231
`
`2/2000 Chana .........
`. 709/231
`6,023,731 A
`~~~~~
`6,032,130 A
`2/2000 A110111 6t 31.
`705/27
`
`.. 709/253
`6,035,341 A
`3/2000 Nunally et al.
`
`6941345 A
`3/2000 Levi et 3L -----
`709/217
`. 709/218
`6,049,823 A
`4/2000 Hwang ..........
`
`.
`6,064,794 A
`5/2000 McLaren et a1.
`386/68
`A
`......... ...... 345/349
`9/2000 Proehl et a1.
`6,118,450
`
`
`
`6,118,817 A
`6,128,653 A
`6,133,920 A
`6,144,375 A
`6,148,304 A
`6,154,771 A
`6,166,314 A
`6,173,317 B1
`6,204,840 B1
`6,215,910 B1
`6,230,172 B1
`6,233,389 B1
`6,363,207 B1*
`6,370,688 B1*
`6,614,843 B1*
`2003/0163824 A1*
`
`9/2000 Wang ......................... 375/240
`10/2000 Del Val et a1.
`.
`709/219
`
`........... 345/354
`10/2000 DeCarmo et 31.
`11/2000 Jain et a1.
`................... 345/302
`
`“/2000 De Vries et 31.
`707/104
`.............. 709/217
`11/2000 Rangan et a1.
`12/2000 Weinstock et a1.
`......... 84/483.1
`
`1/2001 Chaddha et 31.
`......
`709/219
`.......... 345/302
`3/2001 Petelycky et a1.
`4/2001 Chaddha ..................... 382/253
`
`5/2001 Purnaveja et 31.
`.
`707/512
`................. 386/46
`5/2001 Barton et a1.
`3/2002 Duruoz et a1.
`................ 386/68
`
`4/2002 Hejna, Jr.
`.......
`725/101
`......... 375/240.01
`9/2003 Gordon et a1.
`8/2003 Gordon et a1.
`................ 725/90
`
`OTHER PUBLICATIONS
`
`Lynda Hardman et a1., “Multimedia authoring paradigms,” Author—
`ing and Application of Hypermedia—Based User—Inte/faces, IEE
`Colloquium,,The Institution of Electrical Engineers 1995, pp. 8/1-
`8/3.
`John David N. Dionisio and Alfonso F. Cardenas, “A Unified Data
`Model for Representing Multimedia, Timeline, and Simulation
`Data,” IEEE Transactions on Knowledge and Data Engineering,
`V01. 10, N0. 5, Sep./Oct. 1998, pp. 746-767.
`Microsoft Corporation and RealNetworks, Inc., Advanced Stream-
`ing Format (ASF) Specification, Feb. 26, 1998, Public Specification
`Version 1.0, 55 pages.
`Arons, Barry, “SpeechSkimmer: A System for Interactively Skim-
`ming Recorded Speech”, ACM Transactions on Computer-Human,
`V01. 4, N0. 1, pp. 3-38, 1997.
`Internet Reference “An Annotated Bibliography of Interactive
`Speech User Interfaces by Barry Arons” HTTP://barons.www.
`media.mit.edu/people/barons/AronsAnnotatedBibliographyhtml,
`Sep. 15, 1998.
`“GSM Full Rate Speech Transcoding,” ETSI/PT 12, Feb. 1992, pp.
`l-93.
`“Speech codee for the European Mobile Radio System,” P. Vary et
`31,, 1988, pp, 227-230,
`InformediaiInternet References, http://Www.informedia.cs.cmu.
`edu, Dec. 20, 1996.
`
`* cited by examiner
`
`2
`
`

`

`mm
`
`S
`
`,m
`
`US 7,237,254 B1
`
`U.S. Patent
`
`n.m
`
`we
`
`M,Now
`
`N ~
`
`§\
`Dx
`
`o:
`
`Ema
`
`9A522.0
`mX:
`mEEmmbw
`
`520m.
`
`mEEmmEm
`
`moiEma
`
`m_umE:_:_>_
`
`EE26
`
`X:
`
`3
`
`
`

`

`U.S. Patent
`
`US 7,237,254 B1
`
`wEEmoi_uM__538:9?__m.__“mJ__a...._Qamu/x_Emuw>w
`
`
`
`
`
`
`_m_._mw._a02.x8252\Emogoxm_e.__hm2<_moo.__._SmD:n__wW7mm__mm§m__I__2a.__om? l__v9__9mmESmoi._9I._MmomtBEtommomtmE.Nwmmmwfiuommomteczmomgo__.en:5__238:
`
`
`
`
`._.__ND._........................................................._L4L4L4L4L%Lwrdrwrwrw.lfiv2.
` 95$qu_n_82>___l____'IIIIm#5«E:mewmooi_.._,wM.______d20m:__N2.
`
`[3.w.........EQEWSEEWNW:
`
`
`
`$2222,:58:
`
`a
`
`NE.
`
`x5382m3
`
`co=mo=qa<END$382mEEmoE
`mEmhmoiEmon>w¥X:Now09.mm?
`
`Ge’15E!55%$20
`
`4
`
`
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 3 of 9
`
`US 7,237,254 B1
`
`1 02
`
`
`202
`C
`203(
`
`omp03| e
`
`Ini‘r’EZELn
`Media
`
`1 04
`Stream
`
`
`
`
`Audio Time
`
`Modification
`
`Audio Render
`
`
`
`
`Video Time
`
`Modification
`
`
`
`
`
`Video Render
`
`5
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet4 0f9
`
`US 7,237,254 B1
`
`
`
`6
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 5 of 9
`
`US 7,237,254 B1
`
`262
`
`264
`
`Media Screen
`
`MultiMedia Player
`
`270
`
`7
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 6 of 9
`
`US 7,237,254 B1
`
`Client Receives New Playback
`Speed
`
`302
`
`304
`
`
`
`
`Client Communicates New
`
`
`
`Playback Speed to Server
`
`306
`
`308
`
`
`Server Alters Rate of Transfer
`
`Client Begins Modifying Data in
`
`
`
`Accordance with New Playback
`
`
`Speed
`
`of Stream as Necessary to
`Overcompensate for New
`Playback Speed
`
`
`
`
`
`
`Client and Server
`
`
`Resynch ronize
`
`
`
`8
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 7 of 9
`
`US 7,237,254 B1
`
`Client Computer
`
`
`
`
`352
`
`Server Computer
`
`Receive New Playback Speed
`
`
`
`Communicate New Playback
`
`Speed to Server Receive New Playback Speed
`
`from Client
`
`Begin Sending Stream for New
`Playback Speed, Tagging
`Stream Data Packets with the
`
`New Playback Speed
`
`
`
`
`
`
`
`Modify Stream in Accordance
`with Tags of Data Being
`Modified
`
`358
`
`Render Modified Stream
`
`
`
`9
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 8 of 9
`
`US 7,237,254 B1
`
`Client Computer
`
`
`382
`
`Server Computer
`
`
`
`Receive New Playback Speed
`
`
`from Client
`
`Communicate New Playback
`
`Speed to Server Receive New Playback Speed
`
`
`
`Begin Time-Scale Modification
`at New Playback Speed,
`
`Tagging Stream Data Packets
`with New Playback Speed
`
`
`
`
`Receive and Render Data
`Stream
`
`
`Send Stream to Client
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 9 of 9
`
`US 7,237,254 B1
`
`Client Computer
`
`Receive New Playback
`Speed
`
`Server Computer
`
`
`
`
`
`
`Inform Server of New
`
`
`
`Receive New Playback
`Speed
`Playback Speed
`
`
`
`412
`
`Select Stream
`
`
`Corresponding to
`New Playback Speed
`
`Continue to Send
`
`Previous Stream to
`Client
`
`
`Identify Proper
`Location in New
`Stream
`
`414
`
`
`
`
`
`Client Beginning with
`Identified Location
`
`
`Stop Sending Previous
`Stream to Client
`
`418
`
`Send New Stream to
`
`Render Received Stream
`
`
`
`11
`
`11
`
`

`

`US 7,237,254 B1
`
`1
`SEAMLESS SWITCHING BETWEEN
`DIFFERENT PLAYBACK SPEEDS OF
`TIME-SCALE MODIFIED DATA STREAMS
`
`TECHNICAL FIELD
`
`This invention relates to rendering data streams. More
`particularly, the invention relates to switching between dif-
`ferent playback speeds of time-scale modified data streams.
`
`BACKGROUND OF THE INVENTION
`
`Multimedia streamingithe continuous delivery of syn-
`chronized media data like video, audio, text, and anima-
`tioniis a critical link in the digital multimedia revolution.
`Today, streaming media is primarily about video and audio,
`but a richer, broader digital media era is emerging with a
`profound and growing impact on the Internet and digital
`broadcasting.
`Synchronized media refers to multiple media objects that
`share a common timeline. Video and audio are examples of
`synchronized mediaieach is a separate data stream with its
`own data structure, but the two data streams are played back
`in synchronization with each other. Virtually any media type
`can have a timeline. For example, an image object can
`change like an animated .gif file: text can change and move,
`and animation and digital effects happen over time. This
`concept of synchronizing multiple media types is gaining
`greater meaning and currency with the emergence of more
`sophisticated media composition frameworks implied by
`MPEG-4, Dynamic HTML, and other media playback envi-
`ronments.
`
`The term “streaming” is used to indicate that the data
`representing the various media types is provided over a
`network to a client computer on a real-time, as-needed basis,
`rather than being pre-delivered in its entirety before play-
`back. Thus, the client computer renders streaming data as it
`is received from a network server, rather than waiting for an
`entire “file” to be delivered.
`
`The widespread availability of streaming multimedia
`enables a variety of informational content
`that was not
`previously available over the Internet or other computer
`networks. Live content is one significant example of such
`content. Using streaming multimedia, audio, video, or
`audio/visual coverage of noteworthy events can be broadcast
`over the Internet as the events unfold. Similarly, television
`and radio stations can transmit their live content over the
`Internet.
`
`A US. patent application entitled “Multimedia Timeline
`Modification in Networked Client/Server Systems,” filed
`Sep. 15, 1998, Ser. No. 09/153,664, by inventors Anoop
`Gupta and Nosakhare D. Omoigui, describes a system that
`allows a user to vary the playback speed of streaming
`multimedia content using time-scale modification technol-
`ogy. AU.S. patent application entitled “Timeline Correlation
`Between Multiple Timeline-Altered Media Streams,” filed
`Sep. 15, 1998, Ser. No. 09/153,749, by inventors Anoop
`Gupta, Nosakhare D. Omoigui, and Liwei He, describes a
`system that utilizes time-scale modification so that a user
`can vary the speed of streaming content without destroying
`its intelligibility.
`Although such systems allow a user to change the play-
`back speed of streaming multimedia content without
`destroying its intelligibility, such changes are not always
`“seamless”. There may be rather lengthy delays, from the
`user’s perspective, between the user’s request for a new
`playback speed and the actual rendering of the data at that
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`requested speed. Additionally, there may be breaks in the
`data where the user is presented with either a “paused” view
`of the streaming data or no data at all until the system is able
`to render the stream at the requested speed. These problems
`degrade the overall user experience in playing back the
`multimedia content.
`
`The invention described below addresses these problems,
`reducing delays and breaks when switching between differ-
`ent playback speeds of time-scale modified streams.
`
`SUMMARY OF THE INVENTION
`
`is
`In a network environment, multimedia content
`streamed from a server computer to a client computer via the
`network. A user of the client computer can alter the speed at
`which the multimedia content is played, either speeding up
`or slowing down the playback. When the playback speed of
`the multimedia content is changed, the invention seamlessly
`switches between the previous playback speed and the new
`playback speed.
`According to one aspect of the invention, flow control is
`used to provide seamless switching between different play-
`back speeds. The client computer performs time-scale modi-
`fication on data streams received from the server in order to
`
`obtain the playback speed requested by the user. When a new
`playback speed is selected by the user, the server aggres-
`sively refills the client’s data buffers in order to ensure that
`the client has sufficient data to immediately begin time-scale
`modification for the new playback speed.
`According to another aspect of the invention, a data
`stream is transferred form the server to the client as a series
`
`of data packets. The rate at which the packets are transferred
`to the client is based on the playback speed selected by the
`user, and each packet is tagged with the playback speed to
`which it corresponds. In embodiments where the time-scale
`modification is implemented in the client, the client modifies
`the time-scale of the data stream based on these tags. In
`embodiments where the time-scale modification is imple-
`mented in the server, the received time-scale modified data
`is rendered by the client at a playback speed according to the
`tags.
`According to another aspect of the invention, multiple
`different versions of multimedia content are stored at the
`
`server, each version corresponding to a different playback
`speed. When a user selects a new playback speed, a different
`one of these multiple versions is provided from the server to
`the client. During the process of switching versions, the
`server continues to transfer data from the previous version to
`the client until the proper location in the new stream to begin
`transferring is identified. Once the proper location is iden-
`tified, the server stops transferring data from the previous
`version and begins transferring data from the new version.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention is illustrated by way of example
`and not
`limitation in the figures of the accompanying
`drawings. The same numbers are used throughout the figures
`to reference like components and/or features.
`FIG. 1 shows a client/server network system and envi-
`ronment in accordance with the invention.
`
`FIG. 2 shows a general example of a computer that can be
`used as a server or client in accordance with the invention.
`
`FIG. 3 illustrates a system in which timeline modification
`is performed by a client computer.
`FIG. 4 illustrates a system in which multiple versions of
`media streams are stored at a server.
`
`12
`
`12
`
`

`

`US 7,237,254 B1
`
`3
`FIG. 5 shows one implementation of a graphical user
`interface window for a multimedia player.
`FIG. 6 is a flowchart illustrating exemplary steps followed
`in using flow control to seamlessly switch between different
`playback speeds.
`FIG. 7 is a flowchart illustrating exemplary steps followed
`in using stream tagging to seamlessly switch between dif-
`ferent playback speeds.
`FIG. 8 is a flowchart illustrating another example of using
`stream tagging to seamlessly switch between different play-
`back speeds.
`illustrating another example of
`FIG. 9 is a flowchart
`seamlessly switching between different playback speeds.
`
`DETAILED DESCRIPTION
`
`General Network Structure
`
`FIG. 1 shows a client/server network system and envi-
`ronment in accordance with the invention. Generally, the
`system includes one or more network server computers 102,
`and multiple (11) network client computers 104. The com-
`puters communicate with each other over a data communi-
`cations network. The communications network in FIG. 1
`
`comprises a public network 106 such as the Internet. The
`data communications network might also include local-area
`networks and private wide-area networks.
`Multimedia server 102 has access to streaming media
`content in the form of different media streams. These media
`
`streams can be individual media streams (e.g., audio, video,
`graphical, etc.), or alternatively composite media streams
`including multiple such individual streams. Some media
`streams might be stored as files 108 in a database or other file
`storage system, while other media streams 110 might be
`supplied to the server on a “live” basis from other data
`source components
`through dedicated communications
`channels or through the Internet itself.
`Generally, the client computers 104 are responsive to user
`input
`to select or request
`identified media streams.
`In
`response to a request for a media stream, multimedia server
`102 streams the requested media stream to the client 104 in
`accordance with some known format. The client 104 renders
`
`the media stream to produce the content of the stream.
`The invention allows a user to seamlessly switch between
`different playback speeds of time-scale modified media
`streams. For example, a user at a client computer 104 may
`wish to speed up (compressing the time scale) or slow down
`(expanding the time scale) the playback of a media stream
`from multimedia server 102. This switching may involve
`either time-scale modification performed “on the fly” at the
`client and/or the server, or alternatively switching between
`different streams that are two different versions of the same
`
`multimedia content. The invention uses various techniques
`to seamlessly switch between the different playback speeds.
`
`Exemplary Computer Environment
`In the discussion below, the invention will be described in
`the general context of computer-executable instructions,
`such as program modules, being executed by one or more
`conventional personal computers. Generally, program mod-
`ules include routines, programs, objects, components, data
`structures, etc. that perform particular tasks or implement
`particular abstract data types. Moreover, those skilled in the
`art will appreciate that the invention may be practiced with
`other computer system configurations, including hand-held
`devices, multiprocessor systems, microprocessor-based or
`programmable consumer electronics, network PCs, mini-
`computers, mainframe computers, and the like. In a distrib-
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`uted computer environment, program modules may be
`located in both local and remote memory storage devices.
`Alternatively,
`the invention could be implemented in
`hardware or a combination of hardware, software, and/or
`firmware. For example, one or more application specific
`integrated circuits (ASICs) could be programmed to carry
`out the invention.
`
`FIG. 2 shows a general example of a computer 130 that
`can be used as a server or client in accordance with the
`
`invention. Computer 130 is shown as an example of a
`computer that can perform the functions of a server com-
`puter 102 or a client computer 104 of FIG. 1.
`Computer 130 includes one or more processors or pro-
`cessing units 132, a system memory 134, and a bus 136 that
`couples various system components including the system
`memory 134 to processors 132.
`The bus 136 represents one or more of any of several
`types of bus structures, including a memory bus or memory
`controller, a peripheral bus, an accelerated graphics port, and
`a processor or local bus using any of a variety of bus
`architectures. The system memory includes read only
`memory (ROM) 138 and random access memory (RAM)
`140. Abasic input/output system (BIOS) 142, containing the
`basic routines that help to transfer information between
`elements within computer 130, such as during start-up, is
`stored in ROM 138. Computer 130 further includes a hard
`disk drive 144 for reading from and writing to a hard disk,
`not shown, a magnetic disk drive 146 for reading from and
`writing to a removable magnetic disk 148, and an optical
`disk drive 150 for reading from or writing to a removable
`optical disk 152 such as a CD ROM or other optical media.
`The hard disk drive 144, magnetic disk drive 146, and
`optical disk drive 150 are connected to the bus 136 by an
`SCSI interface 154 or some other appropriate interface. The
`drives and their associated computer-readable media provide
`nonvolatile storage of computer readable instructions, data
`structures, program modules and other data for computer
`130. Although the exemplary environment described herein
`employs a hard disk, a removable magnetic disk 148 and a
`removable optical disk 152,
`it should be appreciated by
`those skilled in the art that other types of computer readable
`media which can store data that is accessible by a computer,
`such as magnetic cassettes, flash memory cards, digital
`video disks, random access memories (RAMs) read only
`memories (ROM), and the like, may also be used in the
`exemplary operating environment.
`A number of program modules may be stored on the hard
`disk, magnetic disk 148, optical disk 152, ROM 138, or
`RAM 140, including an operating system 158, one or more
`application programs 160, other program modules 162, and
`program data 164. A user may enter commands and infor-
`mation into computer 130 through input devices such as
`keyboard 166 and pointing device 168. Other input devices
`(not shown) may include a microphone, joystick, game pad,
`satellite dish, scanner, or the like. These and other input
`devices are connected to the processing unit 132 through an
`interface 170 that is coupled to the bus 136. A monitor 172
`or other type of display device is also connected to the bus
`136 via an interface, such as a video adapter 174. In addition
`to the monitor, personal computers typically include other
`peripheral output devices (not shown) such as speakers and
`printers.
`Computer 130 operates in a networked environment using
`logical connections to one or more remote computers, such
`as a remote computer 176. The remote computer 176 may be
`another personal computer, a server, a router, a network PC,
`a peer device or other common network node, and typically
`
`13
`
`13
`
`

`

`US 7,237,254 B1
`
`5
`includes many or all of the elements described above
`relative to computer 130, although only a memory storage
`device 178 has been illustrated in FIG. 2. The logical
`connections depicted in FIG. 2 include a local area network
`(LAN) 180 and a wide area network (WAN) 182. Such
`networking environments are commonplace in offices, enter-
`pri se-wide computer networks, intranets, and the Internet. In
`the described embodiment of the invention, remote com-
`puter 176 executes an Internet Web browser program such as
`the “Internet Explorer” Web browser manufactured and
`distributed by Microsoft Corporation of Redmond, Wash.
`When used in a LAN networking environment, computer
`130 is connected to the local network 180 through a network
`interface or adapter 184. When used in a WAN networking
`environment, computer 130 typically includes a modem 186
`or other means for establishing communications over the
`wide area network 182, such as the Internet. The modem
`186, which may be internal or external, is connected to the
`bus 136 via a serial port interface 156. In a networked
`environment, program modules depicted relative to the
`personal computer 130, or portions thereof, may be stored in
`the remote memory storage device. It will be appreciated
`that the network connections shown are exemplary and other
`means of establishing a communications link between the
`computers may be used.
`Generally, the data processors of computer 130 are pro-
`grammed by means of instructions stored at different times
`in the various computer-readable storage media of the com-
`puter. Programs and operating systems are typically distrib-
`uted, for example, on floppy disks or CD-ROMs. From
`there,
`they are installed or loaded into the secondary
`memory of a computer. At execution, they are loaded at least
`partially into the computer’s primary electronic memory.
`The invention described herein includes these and other
`
`various types of computer-readable storage media when
`such media contain instructions or programs for implement-
`ing the steps described below in conjunction with a micro-
`processor or other data processor. The invention also
`includes the computer itself when programmed according to
`the methods and techniques described below. Furthermore,
`certain sub-components of the computer may be pro-
`grammed to perform the functions and steps described
`below. The invention includes such sub-components when
`they are programmed as described. In addition, the invention
`described herein includes data structures, described below,
`as embodied on various types of memory media.
`For purposes of illustration, programs and other execut-
`able program components such as the operating system are
`illustrated herein as discrete blocks, although it is recog-
`nized that such programs and components reside at various
`times in different storage components of the computer, and
`are executed by the data processor(s) of the computer.
`
`Streaming Media
`In this discussion, the term “composite media stream”
`describes synchronized streaming data that represents a
`segment of multimedia content. The composite media
`stream has a timeline that establishes the speed at which the
`content is rendered. The composite media stream can be
`rendered to produce a plurality of different types of user-
`perceivable media, including synchronized audio or sound,
`video graphics or motion pictures, animation, textual con-
`tent, command script sequences, or other media types that
`convey time-varying information or content in a way that
`can be sensed and perceived by a human. A composite media
`stream comprises a plurality of individual media streams
`representing the multimedia content. Each of the individual
`
`6
`media streams corresponds to and represents a different
`media type and each of the media streams can be rendered
`by a network client to produce a user-perceivable presenta-
`tion using a particular presentation medium. The individual
`media streams have their own timelines, which are synchro-
`nized with each other so that the media streams can be
`
`rendered simultaneously for a coordinated multimedia pre-
`sentation. The individual timelines define the timeline of the
`
`composite stream.
`There are various standards for streaming media content
`and composite media streams. “Advanced Streaming For-
`mat” (ASF) is an example of such a standard, including both
`accepted versions of the standard and proposed standards for
`future adoption. ASF specifies the way in which multimedia
`content is stored, streamed, and presented by the tools,
`servers, and clients of various multimedia vendors. ASF
`provides benefits such as local and network playback, exten-
`sible media types, component download, scaleable media
`types, prioritization of streams, multiple language support,
`environment independence, rich inter-stream relationships,
`and expandability. Further details about ASF are available
`from Microsoft Corporation of Redmond, Wash.
`Regardless of the streaming format used, an individual
`data stream contains a sequence of digital data sets or units
`that are rendered individually, in sequence, to produce an
`image, sound, or some other stimuli that is perceived by a
`human to be continuously varying. For example, an audio
`data stream comprises a sequence of sample values that are
`converted to a pitch and volume to produce continuously
`varying sound. Avideo data stream comprises a sequence of
`digitally-specified graphics frames that are rendered in
`sequence to produce a moving picture.
`Typically, the individual data units of a composite media
`stream are interleaved in a single sequence of data packets.
`Various types of data compression might be used within a
`particular data format to reduce communications bandwidth
`requirements.
`The sequential data units (such as audio sample values or
`video frames) are associated with both delivery times and
`presentation times, relative to an arbitrary start time. The
`delivery time of a data unit indicates when the data unit
`should be delivered to a rendering client. The presentation
`time indicates when the value should be actually rendered.
`Normally,
`the delivery time of a data unit precedes its
`presentation time.
`The presentation times determine the actual speed of
`playback. For data streams representing actual events or
`performances, the presentation times correspond to the rela-
`tive times at which the data samples were actually recorded.
`The presentation times of the various different individual
`data streams are consistent with each other so that the
`
`streams remain coordinated and synchronized during play-
`back.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`Multimedia Time-Scale Modification
`
`A network client 104 of FIG. 1 can accept a speed
`designation from a user. In the illustrated example, the speed
`designation is a speed factor relative to the original or
`default playback speed of the selected multimedia stream.
`For example, a speed factor of 1.2 indicates that the com-
`posite media stream is to be rendered at 1.2 times its original
`or default speed, thereby achieving time compression. A
`speed factor of 0.8 indicates that the composite media stream
`is to be rendered at 0.8 times its original or default speed,
`thereby achieving time expansion.
`In response to the speed designation from the user, the
`system modifies the timelines of the individual media
`
`60
`
`65
`
`14
`
`14
`
`

`

`US 7,237,254 B1
`
`7
`streams of the composite media stream, while keeping the
`timelines synchronized with each other and while maintain-
`ing the original pitch of any audio produced from audio
`streams. In one embodiment of the invention, such timeline
`modification is performed by the network client. In other
`embodiments of the invention, the timeline modification can
`be performed at
`the network server, before the media
`streams are streamed to the network client.
`
`the timeline of the
`Timeline modification changes
`received data streams in accordance with the user speed
`designation to achieve either time compression or time
`expansion. With some types of media, such as video streams,
`this involves either omitting selected frames or modifying
`the presentation times of the individual data units or video
`frames. In other cases, such as with audio streams,
`the
`time-modification is more difi‘icultisimply changing the
`presentation times would alter the pitch of the or