USOO7237254B1
`
`(12) United States Patent
`Omoigui
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,237,254 B1
`Jun. 26, 2007
`
`(54) SEAMLESS SWITCHING BETWEEN
`DIFFERENT PLAYBACK SPEEDS OF
`TIME-SCALEMODIFIED DATA STREAMS
`
`(75) Inventor: Nosakhare D. Omoigui, Redmond, WA
`(US)
`
`(73) Assignee: Microsoft Corporation, Redmond, WA
`(US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/539,170
`
`(22) Filed:
`
`Mar. 29, 2000
`
`(51) Int. Cl.
`(2006.01)
`HO)4N 7/173
`(52) U.S. Cl. ......................................... 725/94; 709/231
`(58) Field of Classification Search .................. 725/88,
`725/102, 93, 94
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`9/1996 Gough et al. ............... 395/155
`5,559,942 A
`5,566,175 A 10/1996 Davis ...........
`370.84
`5,574,724. A 1 1/1996 Bales et al. ................ 370/681
`5,606,359 A * 2/1997 Youden et al. ................ 725/88
`5,614,940 A
`3/1997 Cobbley et al. ............... 348.7
`5,617,423. A
`4/1997 Li et al. ..................... 370/426
`5,623,690 A
`4, 1997 Palmer et al. .............. 395/806
`5,625,405 A
`4/1997 Dulac et al. .................. 348.7
`5,640,320 A
`6/1997 Jackson et al. ............. 364/192
`
`EP
`EP
`EP
`EP
`EP
`EP
`WO
`WO
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`0605115
`T 1994
`O653884
`5, 1995
`O 669 587 A2
`8, 1995
`O676898
`10, 1995
`O746,158
`12/1996
`O 812 112 A2 10, 1997
`WO 94/O1964
`1, 1994
`WO 98/37698
`8, 1998
`
`OTHER PUBLICATIONS
`
`H.J. Chen et al., “A Scalable Video-on-Demand Service for the
`Provision of VCR-Like Functions.” IEEE, May 15, 1995, pp. 65-72.
`(Continued)
`
`Primary Examiner Andrew Y. Koenig
`(74) Attorney, Agent, or Firm-Perkins Coie LLP
`(57)
`ABSTRACT
`
`6, 1990 Isle et al. ................... 364,513
`4,931,950 A
`9, 1991 Golestani ..................... 370/60
`5,050,161 A
`6/1992 Beitel et al. ................
`5,119.474. A
`5,274,758 A 12/1993 Beitel et al. .
`... 395,154
`5,309,562 A
`5/1994 Li ..............
`... 395/200
`5,313,454 A
`5/1994 Bustini et al. ................ 37Of 13
`5,341,474 A
`8, 1994 Gelman et al. .
`... 395/200
`In a network environment, multimedia content is streamed
`5,414,455 A
`5, 1995 Hooper et al. .............. 348.7
`from a server computer to a client computer via the network.
`5.434,848 A
`7, 1995 Chimento, Jr. et al. ....... 37Of 17
`A user of the client computer can alter the speed at which the
`5,455,910 A 10, 1995 Johnson et al. ............. 395/650
`multimedia content is played; either speeding up or slowing
`5,481,542 A
`1/1996 Logston et al. ............ 370,942
`down the playback. The client seamlessly switches between
`5.490,252 A
`2f1996 Macera et al. ......... 395/200.01
`5,504,744 A
`4, 1996 Ad
`tal
`370/60.1
`the different playback speeds, reducing breaks and/or delays
`5.519.701 A
`5, 1996 (SN
`al - - - - - - - - - - - - - 370,60.
`between the time the user selects the new playback speed
`5,521,630 A
`5/1996 Chen et al. .................... 348.7 th the ultimedia content begins being played
`5,533,021 A
`7/1996 Branstad et al. ........... 370/60.1
`ack at the new speed.
`5,537,408 A
`7, 1996 Branstad et al. .............. 37Of79
`5,541.955 A
`7, 1996 Jacobsmeyer ............... 375.222
`
`16 Claims, 9 Drawing Sheets
`
`2
`
`Composite
`Media
`
`
`
`
`
`
`
`Audio Render
`
`Wided Render
`
`Petitioners' Exhibit 1032
`Page 0001
`
`

`

`US 7,237,254 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`9, 1997 Mauldin et al. ............ 395/778
`5,664.227 A
`5,692.213 A 1 1/1997 Goldberg et al. ........... 395/806
`5,699.474. A * 12/1997 Suzuki et al. ................. 386.68
`5,717,691 A
`2/1998 Dighe et al. ...
`... 370/401
`5,717,869 A
`2, 1998 Moran et al. ..
`... 395,339
`5,719,786 A
`2, 1998 Nelson et al. .............. 364,514
`5,721,829. A
`2, 1998 Dunn et al. .......
`... 395/20049
`5,742,347 A
`4, 1998 Kandlur et al. ............. 348/426
`5,768,533 A
`6, 1998 Ran ..............
`395/2OO.77
`5,786,814 A
`7, 1998 Moran et al. ............... 345,328
`5,794.210 A
`8/1998 Goldhaber et al. ........... 705/14
`5,794,249 A
`8, 1998 Orsolini et al. ...
`707/104
`5,799,292 A
`8/1998 Hekmatpour ..
`... 706.11
`5,801,685. A
`9, 1998 Miller et al. ...
`... 345,302
`5,808,662 A
`9/1998 Kinney et al. ....
`... 348.15
`5,818,510 A 10/1998 Cobbley et al. ............... 348.7
`5,822,537 A 10, 1998 Katseff et al. ......... 395.200.61
`5,828,848 A 10, 1998 MacCormack et al. 395/200.77
`5,835,495 A 11/1998 Ferriere ...................... 370/465
`5,835,667 A 1 1/1998 Wactlar et al. ................ 386.96
`5,838,906 A 11/1998 Doyle et al. ...
`395.200.32
`5,859,641 A
`1/1999 Cave .......................... 345.348
`5,864,682 A
`1/1999 Porter et al. ...
`... 395/2OO.77
`5,870,755 A
`2, 1999 Stevens et al. ............. TO7 104
`5,873,735 A
`2f1999 Yamada et al.
`... 434/316
`5,892.506 A
`4/1999 Hermanson .
`345,302
`5,903,673 A
`5/1999 Wang et al. ...
`... 382,236
`5,918,002 A
`6, 1999 Klemets et al. ........ 395, 18216
`5,930,787 A
`7, 1999 Minakuchi et al. ............ 707/4
`5,953,506 A
`9, 1999 Kalra et al. ............ 395.200.61
`5,956,716 A
`9, 1999 Kenner et al. ................ 707/10
`5,963,202 A * 10/1999 Polish ...........
`715/723
`5,995,941 A 11/1999 Maquire et al. ..
`... TOS/10
`5.999,979 A 12/1999 Vellanki et al. ...... ... 709,232
`6,006,241 A 12/1999 Purnaveja et al. .......... 707/512
`6,014,706 A
`1/2000 Cannon et al. ...
`... 709,231
`6,023,731 A
`2/2000 Chawla ...................... TO9,231
`6,032,130 A
`2/2000 Alloul et al. ................. 705/27
`6,035,341 A
`3/2000 Nunally et al. ............. 709/253
`6,041,345 A
`3/2000 Levi et al. .................. 709/217
`6,049,823. A
`4/2000 Hwang ....................... 709/218
`6,064,794. A
`5, 2000 McLaren et al. .
`... 386.68
`9, 2000 Proehl et al. ......... - - - - - - 345,349
`6,118,450
`A
`
`
`
`9/2000 Wang ......................... 375,240
`6,118,817 A
`6,128,653 A 10/2000 Del Val et al. .
`... TO9,219
`6,133,920 A 10/2000 DeCarmo et al. ........... 345,354
`6,144,375 A 1 1/2000 Jain et al. ................... 345,302
`6,148,304. A 11/2000 De Vries et al.
`... TO7 104
`6,154,771 A 1 1/2000 Rangan et al. .............. 709/217
`6, 166,314. A 12/2000 Weinstock et al. ......... 84.483.1
`6,173,317 B1
`1/2001 Chaddha et al. ...
`TO9,219
`6,204.840 B1
`3/2001 Petelycky et al. .......... 345,302
`6,215,910 B1
`4/2001 Chaddha ..................... 382/253
`6,230,172 B1
`5/2001 Purnaveja et al. .
`707/512
`6,233,389 B1
`5/2001 Barton et al. ................. 386, 46
`6,363,207 B1 * 3/2002 Duruoz et al. ................ 386.68
`6,370,688 B1 * 4/2002 Hejna, Jr. ...
`... 725,101
`6,614,843 B1* 9/2003 Gordon et al. ......... 375,240.01
`2003/0163824 A1* 8, 2003 Gordon et al. ................ 725/90
`
`OTHER PUBLICATIONS
`
`Lynda Hardman et al., “Multimedia authoring paradigms. Author
`ing and Application of Hypermedia-Based User-Interfaces, IEE
`Colloquium,...The Institution of Electrical Engineers 1995, pp. 871
`8.3.
`John David N. Dionisio and Alfonso F. Cardenas, “A Unified Data
`Model for Representing Multimedia, Timeline, and Simulation
`Data.” IEEE Transactions on Knowledge and Data Engineering,
`vol. 10, No. 5, Sep./Oct. 1998, pp. 746-767.
`Microsoft Corporation and RealNetworks, Inc., Advanced Stream
`ing Format (ASF) Specification, Feb. 26, 1998, Public Specification
`Version 1.0, 55 pages.
`Arons, Barry, “SpeechSkimmer: A System for Interactively Skim
`ming Recorded Speech'. ACM Transactions on Computer-Human,
`vol. 4, No. 1, pp. 3-38, 1997.
`Internet Reference "An Annotated Bibliography of Interactive
`Speech User Interfaces by Barry Arons' HTTP://barons.www.
`media.mit.edu/people?barons. Arons AnnotatedBibliography.html,
`Sep. 15, 1998.
`“GSM Full Rate Speech Transcoding.” ETSI/PT 12, Feb. 1992, pp.
`1-93.
`“Speech codee for the European Mobile Radio System.” P. Vary et
`al., 1988, pp. 227-230.
`Informedia Internet References, http://www.informedia.cs.cmu.
`edu, Dec. 20, 1996.
`
`* cited by examiner
`
`Petitioners' Exhibit 1032
`Page 0002
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 1 of 9
`
`US 7,237,254 B1
`US 7,237,254 B1
`
`108
`
`102
`
`StreamingDataFiles
`
`Multimedia
`
`Server
`
`
`
`
`
`
`
`
`
`
`—
`=
`=
`c
`AY
`
`xt3
`-
`
`StreamingData
`
`Y.
`me
`.
`
`s
`
`SS
`N
`
`©
`~
`c
`2
`
`3
`-
`
`Petitioners’ Exhibit 1032
`Page 0003
`
`Petitioners' Exhibit 1032
`Page 0003
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 2 of 9
`
`US 7,237,254 B1
`
`cLb
`
`!|
`
`
`
`O8LYIOMION
`
`
`
`ealy|e007]
`
`uoqeoyddy
`
`swes6old
`
`c8l
`
`WOMION
`
`(Nesters
`
`
`
`BaySpiWepo|y
`
`
`
`
` OLL!i|:9Eli|||09}|;|i(8cT:{Jajdepywashks !]—|soepayu||HodSOBHotresoRO:asnoy|Buyesado|!ioapiA|!.|.!pljunBuisssd0ig
`
`HOMIEN|{IES||ineogkey
`pieogheyvol
`
`SORLOIU|ISOS
` sajnpol
`wesBolduoneajddy|BuryeiadoJBUIO
`
`
`sweibog|uoneoyddy:|
`
`POL
`
`eyeqgweibold
`
`Petitioners’ Exhibit 1032
`Page 0004
`
`Petitioners' Exhibit 1032
`Page 0004
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 3 of 9
`
`US 7,237,254 B1
`
`102
`
`2O2
`
`in Ron
`
`104
`
`C
`omposite
`Media
`Stream
`
`Audio Time
`Modification
`
`Audio Render
`
`
`
`
`
`
`
`
`
`Video Time
`Modification
`
`Video Render
`
`
`
`
`
`
`
`
`
`Petitioners' Exhibit 1032
`Page 0005
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jun. 26, 2007
`Jun. 26, 2007
`
`Sheet 4 of 9
`Sheet 4 of 9
`
`US 7,237,254 B1
`US 7,237,254 B1
`
`
`
`
`ee eee
` Client
`
`— ee ee ee ee EE EE EE EE
`
`Petitioners’ Exhibit 1032
`Page 0006
`
`Petitioners' Exhibit 1032
`Page 0006
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 5 Of 9
`
`US 7,237,254 B1
`
`
`
`262
`
`MultiMedia Player
`
`264
`
`Media Screen
`
`270
`
`Petitioners' Exhibit 1032
`Page 0007
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 6 of 9
`
`US 7,237,254 B1
`
`
`
`306
`
`302
`
`304
`
`Client Receives New Playback
`Speed
`
`Client Communicates New
`Playback Speed to Server
`
`Client Begins Modifying Data in
`Accordance with New Playback
`Speed
`
`
`
`
`
`Server Alters Rate of Transfer
`of Stream as Necessary to
`Overcompensate for New
`Playback Speed
`
`3O8
`
`Client and Server
`Resynchronize
`
`
`
`
`
`Petitioners' Exhibit 1032
`Page 0008
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 7 Of 9
`
`US 7,237,254 B1
`
`Client Computer
`
`352
`
`Server Computer
`
`Receive New Playback Speed
`
`Communicate New Playback
`Speed to Server
`
`Receive New Playback Speed
`from Client
`
`
`
`Begin Sending Stream for New
`Playback Speed, Tagging
`Stream Data Packets with the
`New Playback Speed
`
`Modify Stream in Accordance
`with Tags of Data Being
`Modified
`
`358
`
`Render Modified Stream
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioners' Exhibit 1032
`Page 0009
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 8 of 9
`
`US 7,237,254 B1
`
`Client Computer
`
`382
`
`Server Computer
`
`Receive New Playback Speed
`
`Communicate New Playback
`Speed to Server
`
`Receive New Playback Speed
`from Client
`
`Begin Time-Scale Modification
`at New Playback Speed,
`Tagging Stream Data Packets
`with New Playback Speed
`
`Send Stream to Client
`
`
`
`
`
`Receive and Render Data
`Stream
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioners' Exhibit 1032
`Page 0010
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 9 Of 9
`
`US 7,237,254 B1
`
`Client Computer
`
`Server Computer
`
`Receive New Playback
`Speed
`
`inform Server of New
`Playback Speed
`
`Receive New Playback
`Speed
`
`
`
`
`
`
`
`
`
`
`
`
`
`Continue to Send
`Previous Stream to
`Client
`
`412
`
`Select Stream
`Corresponding to
`New Playback Speed
`
`414
`
`Identify Proper
`Location in New
`Stream
`
`Stop Sending Previous
`Stream to Client
`
`418
`
`
`
`Send New Stream to
`Client Beginning with
`ldentified Location
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Render Received Stream
`
`Petitioners' Exhibit 1032
`Page 0011
`
`

`

`US 7,237,254 B1
`
`1.
`SEAMLESS SWITCHING BETWEEN
`DIFFERENT PLAYBACK SPEEDS OF
`TIME-SCALEMODIFIED DATA STREAMS
`
`TECHNICAL FIELD
`
`This invention relates to rendering data streams. More
`particularly, the invention relates to switching between dif
`ferent playback speeds of time-scale modified data streams.
`
`BACKGROUND OF THE INVENTION
`
`Multimedia streaming the continuous delivery of syn
`chronized media data like Video, audio, text, and anima
`tion is a critical link in the digital multimedia revolution.
`Today, streaming media is primarily about video and audio,
`but a richer, broader digital media era is emerging with a
`profound and growing impact on the Internet and digital
`broadcasting.
`Synchronized media refers to multiple media objects that
`share a common timeline. Video and audio are examples of
`synchronized media—each is a separate data stream with its
`own data structure, but the two data streams are played back
`in Synchronization with each other. Virtually any media type
`can have a timeline. For example, an image object can
`change like an animated .gif file: text can change and move,
`and animation and digital effects happen over time. This
`concept of synchronizing multiple media types is gaining
`greater meaning and currency with the emergence of more
`Sophisticated media composition frameworks implied by
`MPEG-4, Dynamic HTML, and other media playback envi
`rOnmentS.
`The term “streaming is used to indicate that the data
`representing the various media types is provided over a
`network to a client computer on a real-time, as-needed basis,
`rather than being pre-delivered in its entirety before play
`back. Thus, the client computer renders streaming data as it
`is received from a network server, rather than waiting for an
`entire “file to be delivered.
`The widespread availability of streaming multimedia
`enables a variety of informational content that was not
`previously available over the Internet or other computer
`networks. Live content is one significant example of Such
`content. Using streaming multimedia, audio, video, or
`audio/visual coverage of noteworthy events can be broadcast
`over the Internet as the events unfold. Similarly, television
`and radio stations can transmit their live content over the
`Internet.
`A U.S. patent application entitled “Multimedia Timeline
`Modification in Networked Client/Server Systems.” filed
`Sep. 15, 1998, Ser. No. 09/153,664, by inventors Anoop
`Gupta and Nosakhare D. Omoigui, describes a system that
`allows a user to vary the playback speed of streaming
`multimedia content using time-scale modification technol
`ogy. A U.S. patent application entitled “Timeline Correlation
`Between Multiple Timeline-Altered Media Streams, filed
`Sep. 15, 1998, Ser. No. 09/153,749, by inventors Anoop
`Gupta, Nosakhare D. Omoigui, and Liwei He, describes a
`system that utilizes time-scale modification so that a user
`can vary the speed of streaming content without destroying
`its intelligibility.
`Although Such systems allow a user to change the play
`back speed of streaming multimedia content without
`destroying its intelligibility, Such changes are not always
`"seamless”. There may be rather lengthy delays, from the
`user's perspective, between the user's request for a new
`playback speed and the actual rendering of the data at that
`
`2
`requested speed. Additionally, there may be breaks in the
`data where the user is presented with either a “paused' view
`of the streaming data or no data at all until the system is able
`to render the stream at the requested speed. These problems
`degrade the overall user experience in playing back the
`multimedia content.
`The invention described below addresses these problems,
`reducing delays and breaks when switching between differ
`ent playback speeds of time-scale modified streams.
`
`SUMMARY OF THE INVENTION
`
`In a network environment, multimedia content is
`streamed from a server computer to a client computer via the
`network. A user of the client computer can alter the speed at
`which the multimedia content is played, either speeding up
`or slowing down the playback. When the playback speed of
`the multimedia content is changed, the invention seamlessly
`switches between the previous playback speed and the new
`playback speed.
`According to one aspect of the invention, flow control is
`used to provide seamless switching between different play
`back speeds. The client computer performs time-scale modi
`fication on data streams received from the server in order to
`obtain the playback speed requested by the user. When a new
`playback speed is selected by the user, the server aggres
`sively refills the client’s data buffers in order to ensure that
`the client has sufficient data to immediately begin time-scale
`modification for the new playback speed.
`According to another aspect of the invention, a data
`stream is transferred form the server to the client as a series
`of data packets. The rate at which the packets are transferred
`to the client is based on the playback speed selected by the
`user, and each packet is tagged with the playback speed to
`which it corresponds. In embodiments where the time-scale
`modification is implemented in the client, the client modifies
`the time-scale of the data stream based on these tags. In
`embodiments where the time-scale modification is imple
`mented in the server, the received time-scale modified data
`is rendered by the client at a playback speed according to the
`tags.
`According to another aspect of the invention, multiple
`different versions of multimedia content are stored at the
`server, each version corresponding to a different playback
`speed. When a user selects a new playback speed, a different
`one of these multiple versions is provided from the server to
`the client. During the process of Switching versions, the
`server continues to transfer data from the previous version to
`the client until the proper location in the new stream to begin
`transferring is identified. Once the proper location is iden
`tified, the server stops transferring data from the previous
`version and begins transferring data from the new version.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention is illustrated by way of example
`and not limitation in the figures of the accompanying
`drawings. The same numbers are used throughout the figures
`to reference like components and/or features.
`FIG. 1 shows a client/server network system and envi
`ronment in accordance with the invention.
`FIG. 2 shows a general example of a computer that can be
`used as a server or client in accordance with the invention.
`FIG. 3 illustrates a system in which timeline modification
`is performed by a client computer.
`FIG. 4 illustrates a system in which multiple versions of
`media streams are stored at a server.
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Petitioners' Exhibit 1032
`Page 0012
`
`

`

`US 7,237,254 B1
`
`3
`FIG. 5 shows one implementation of a graphical user
`interface window for a multimedia player.
`FIG. 6 is a flowchart illustrating exemplary steps followed
`in using flow control to seamlessly switch between different
`playback speeds.
`FIG. 7 is a flowchart illustrating exemplary steps followed
`in using stream tagging to seamlessly Switch between dif
`ferent playback speeds.
`FIG. 8 is a flowchart illustrating another example of using
`stream tagging to seamlessly switch between different play
`back speeds.
`FIG. 9 is a flowchart illustrating another example of
`seamlessly Switching between different playback speeds.
`
`DETAILED DESCRIPTION
`
`10
`
`15
`
`4
`uted computer environment, program modules may be
`located in both local and remote memory storage devices.
`Alternatively, the invention could be implemented in
`hardware or a combination of hardware, software, and/or
`firmware. For example, one or more application specific
`integrated circuits (ASICs) could be programmed to carry
`out the invention.
`FIG. 2 shows a general example of a computer 130 that
`can be used as a server or client in accordance with the
`invention. Computer 130 is shown as an example of a
`computer that can perform the functions of a server com
`puter 102 or a client computer 104 of FIG. 1.
`Computer 130 includes one or more processors or pro
`cessing units 132, a system memory 134, and a bus 136 that
`couples various system components including the system
`memory 134 to processors 132.
`The bus 136 represents one or more of any of several
`types of bus structures, including a memory bus or memory
`controller, a peripheral bus, an accelerated graphics port, and
`a processor or local bus using any of a variety of bus
`architectures. The system memory includes read only
`memory (ROM) 138 and random access memory (RAM)
`140. A basic input/output system (BIOS) 142, containing the
`basic routines that help to transfer information between
`elements within computer 130. Such as during start-up, is
`stored in ROM 138. Computer 130 further includes a hard
`disk drive 144 for reading from and writing to a hard disk,
`not shown, a magnetic disk drive 146 for reading from and
`writing to a removable magnetic disk 148, and an optical
`disk drive 150 for reading from or writing to a removable
`optical disk 152 such as a CD ROM or other optical media.
`The hard disk drive 144, magnetic disk drive 146, and
`optical disk drive 150 are connected to the bus 136 by an
`SCSI interface 154 or some other appropriate interface. The
`drives and their associated computer-readable media provide
`nonvolatile storage of computer readable instructions, data
`structures, program modules and other data for computer
`130. Although the exemplary environment described herein
`employs a hard disk, a removable magnetic disk 148 and a
`removable optical disk 152, it should be appreciated by
`those skilled in the art that other types of computer readable
`media which can store data that is accessible by a computer,
`Such as magnetic cassettes, flash memory cards, digital
`Video disks, random access memories (RAMS) read only
`memories (ROM), and the like, may also be used in the
`exemplary operating environment.
`A number of program modules may be stored on the hard
`disk, magnetic disk 148, optical disk 152, ROM 138, or
`RAM 140, including an operating system 158, one or more
`application programs 160, other program modules 162, and
`program data 164. A user may enter commands and infor
`mation into computer 130 through input devices such as
`keyboard 166 and pointing device 168. Other input devices
`(not shown) may include a microphone, joystick, game pad,
`satellite dish, scanner, or the like. These and other input
`devices are connected to the processing unit 132 through an
`interface 170 that is coupled to the bus 136. A monitor 172
`or other type of display device is also connected to the bus
`136 via an interface, such as a video adapter 174. In addition
`to the monitor, personal computers typically include other
`peripheral output devices (not shown) such as speakers and
`printers.
`Computer 130 operates in a networked environment using
`logical connections to one or more remote computers, such
`as a remote computer 176. The remote computer 176 may be
`another personal computer, a server, a router, a network PC,
`a peer device or other common network node, and typically
`
`25
`
`30
`
`35
`
`40
`
`50
`
`General Network Structure
`FIG. 1 shows a client/server network system and envi
`ronment in accordance with the invention. Generally, the
`system includes one or more network server computers 102.
`and multiple (n) network client computers 104. The com
`puters communicate with each other over a data communi
`cations network. The communications network in FIG. 1
`comprises a public network 106 such as the Internet. The
`data communications network might also include local-area
`networks and private wide-area networks.
`Multimedia server 102 has access to streaming media
`content in the form of different media streams. These media
`streams can be individual media streams (e.g., audio, video,
`graphical, etc.), or alternatively composite media streams
`including multiple such individual streams. Some media
`streams might be stored as files 108 in a database or other file
`storage system, while other media streams 110 might be
`supplied to the server on a “live' basis from other data
`Source components through dedicated communications
`channels or through the Internet itself.
`Generally, the client computers 104 are responsive to user
`input to select or request identified media streams. In
`response to a request for a media stream, multimedia server
`102 streams the requested media stream to the client 104 in
`accordance with some known format. The client 104 renders
`the media stream to produce the content of the stream.
`The invention allows a user to seamlessly switch between
`different playback speeds of time-scale modified media
`45
`streams. For example, a user at a client computer 104 may
`wish to speed up (compressing the time scale) or slow down
`(expanding the time scale) the playback of a media stream
`from multimedia server 102. This switching may involve
`either time-scale modification performed “on the fly' at the
`client and/or the server, or alternatively switching between
`different streams that are two different versions of the same
`multimedia content. The invention uses various techniques
`to seamlessly switch between the different playback speeds.
`Exemplary Computer Environment
`In the discussion below, the invention will be described in
`the general context of computer-executable instructions,
`Such as program modules, being executed by one or more
`conventional personal computers. Generally, program mod
`ules include routines, programs, objects, components, data
`60
`structures, etc. that perform particular tasks or implement
`particular abstract data types. Moreover, those skilled in the
`art will appreciate that the invention may be practiced with
`other computer system configurations, including hand-held
`devices, multiprocessor systems, microprocessor-based or
`programmable consumer electronics, network PCs, mini
`computers, mainframe computers, and the like. In a distrib
`
`55
`
`65
`
`Petitioners' Exhibit 1032
`Page 0013
`
`

`

`5
`includes many or all of the elements described above
`relative to computer 130, although only a memory storage
`device 178 has been illustrated in FIG. 2. The logical
`connections depicted in FIG. 2 include a local area network
`(LAN) 180 and a wide area network (WAN) 182. Such
`networking environments are commonplace in offices, enter
`prise-wide computer networks, intranets, and the Internet. In
`the described embodiment of the invention, remote com
`puter 176 executes an Internet Web browser program such as
`the “Internet Explorer Web browser manufactured and
`distributed by Microsoft Corporation of Redmond, Wash.
`When used in a LAN networking environment, computer
`130 is connected to the local network 180 through a network
`interface or adapter 184. When used in a WAN networking
`environment, computer 130 typically includes a modem 186
`or other means for establishing communications over the
`wide area network 182, such as the Internet. The modem
`186, which may be internal or external, is connected to the
`bus 136 via a serial port interface 156. In a networked
`environment, program modules depicted relative to the
`personal computer 130, or portions thereof, may be stored in
`the remote memory storage device. It will be appreciated
`that the network connections shown are exemplary and other
`means of establishing a communications link between the
`computers may be used.
`Generally, the data processors of computer 130 are pro
`grammed by means of instructions stored at different times
`in the various computer-readable storage media of the com
`puter. Programs and operating systems are typically distrib
`uted, for example, on floppy disks or CD-ROMs. From
`there, they are installed or loaded into the secondary
`memory of a computer. At execution, they are loaded at least
`partially into the computer's primary electronic memory.
`The invention described herein includes these and other
`various types of computer-readable storage media when
`Such media contain instructions or programs for implement
`ing the steps described below in conjunction with a micro
`processor or other data processor. The invention also
`includes the computer itself when programmed according to
`the methods and techniques described below. Furthermore,
`certain Sub-components of the computer may be pro
`grammed to perform the functions and steps described
`below. The invention includes such sub-components when
`they are programmed as described. In addition, the invention
`described herein includes data structures, described below,
`as embodied on various types of memory media.
`For purposes of illustration, programs and other execut
`able program components such as the operating system are
`illustrated herein as discrete blocks, although it is recog
`nized that Such programs and components reside at various
`times in different storage components of the computer, and
`are executed by the data processor(s) of the computer.
`Streaming Media
`In this discussion, the term "composite media stream”
`describes synchronized streaming data that represents a
`segment of multimedia content. The composite media
`stream has a timeline that establishes the speed at which the
`content is rendered. The composite media stream can be
`rendered to produce a plurality of different types of user
`perceivable media, including synchronized audio or Sound,
`Video graphics or motion pictures, animation, textual con
`tent, command Script sequences, or other media types that
`convey time-varying information or content in a way that
`can be sensed and perceived by a human. A composite media
`stream comprises a plurality of individual media streams
`representing the multimedia content. Each of the individual
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 7,237,254 B1
`
`10
`
`15
`
`6
`media streams corresponds to and represents a different
`media type and each of the media streams can be rendered
`by a network client to produce a user-perceivable presenta
`tion using a particular presentation medium. The individual
`media streams have their own timelines, which are synchro
`nized with each other so that the media streams can be
`rendered simultaneously for a coordinated multimedia pre
`sentation. The individual timelines define the timeline of the
`composite stream.
`There are various standards for streaming media content
`and composite media streams. “Advanced Streaming For
`mat’ (ASF) is an example of such a standard, including both
`accepted versions of the standard and proposed standards for
`future adoption. ASF specifies the way in which multimedia
`content is stored, streamed, and presented by the tools,
`servers, and clients of various multimedia vendors. ASF
`provides benefits such as local and network playback, exten
`sible media types, component download, Scaleable media
`types, prioritization of streams, multiple language Support,
`environment independence, rich inter-stream relationships,
`and expandability. Further details about ASF are available
`from Microsoft Corporation of Redmond, Wash.
`Regardless of the streaming format used, an individual
`data stream contains a sequence of digital data sets or units
`that are rendered individually, in sequence, to produce an
`image, Sound, or some other stimuli that is perceived by a
`human to be continuously varying. For example, an audio
`data stream comprises a sequence of sample values that are
`converted to a pitch and Volume to produce continuously
`varying Sound. A video data stream comprises a sequence of
`digitally-specified graphics frames that are rendered in
`sequence to produce a moving picture.
`Typically, the individual data units of a composite media
`stream are interleaved in a single sequence of data packets.
`Various types of data compression might be used within a
`particular data format to reduce communications bandwidth
`requirements.
`The sequential data units (such as audio sample values or
`video frames) are associated with both delivery times and
`presentation times, relative to an arbitrary start time. The
`delivery time of a data unit indicates when the data unit
`should be delivered to a rendering client. The presentation
`time indicates when the value should be actually rendered.
`Normally, the delivery time of a data unit precedes its
`presentation time.
`The presentation times determine the actual speed of
`playback. For data streams representing actual events or
`performances, the presentation times correspond to the rela
`tive times at which the data samples were actually recorded.
`The presentation times of the various different individual
`data streams are consistent with each other so that the
`streams remain coordinated and synchronized during play
`back.
`
`Multimedia Time-Scale Modification
`A network client 104 of FIG. 1 can accept a speed
`designation