(12) United States Patent
`Omoigui
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,149,359 B1
`Dec. 12, 2006
`
`US007 149359B1
`
`(54) SEARCHING AND RECORDING MEDIA
`STREAMS
`
`(75) Inventor: Nosakhare D. Omoigui, Redmond, WA
`(US)
`(73) Assignee: Microsoft Corporation, Redmond, WA
`(US)
`0
`-
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(*) Notice:
`
`(21) Appl. No.: 09/465,530
`
`(22) Filed:
`
`Dec. 16, 1999
`
`(51) Int. Cl.
`G06K 9/68
`G06F 5/00
`GO6K 9/OO
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`(52) U.S. C. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 382/219; 345/3O2
`
`(58) Field of Classification Search ............. 715/500.1,
`715/501.1, 709/231, 203: 345/723, 302;
`707/3: 382/218 220,305
`See application file for complete search history.
`References Cited
`
`(56)
`
`7, 1996 Branstad et al. ........... 370,601
`5,533,021 A
`5.535,063 A * 7/1996 Lamming ...................... 360/4
`5,537,408 A
`7, 1996 Branstad et al. .............. 370/79
`5,541.955 A
`7, 1996 Jacobsmeyer ............... 375.222
`5,559,942 A
`9, 1996 Gough et al. ............... 395/155
`5,566,175 A 10, 1996 Davis .......................... 370.84
`3. A yogastr 'i'
`5,617.423 A
`4, 1997 Li et al. ..................... 370/426
`5,623,690 A
`4, 1997 Palmer et al. .............. 395/806
`5,625,405. A
`4, 1997 Dulac et al. .................. 348.7
`5,640,320 A
`6/1997 Jackson et al. ............. 364/192
`5,664.227 A
`9, 1997 Mauldin et al. ............ 395/778
`5,692.213 A
`1 1/1997 Goldberg et al. ........... 395/806
`5,717,691 A
`2/1998 Dighe et al. ................ 370/401
`5,717,869 A
`2, 1998 Moran et al. ............... 395,339
`5,719,786 A
`2, 1998 Nelson et al. .............. 364,514
`5,721,829 A
`2, 1998 Dunn et al. ............ 395.200.49
`5,742,347 A
`4, 1998 Kandlur et al. ............. 348/426
`5,768,533 A
`6/1998 Ran ...................... 395/2OO.77
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`
`0605115
`
`T 1994
`
`(Continued)
`OTHER PUBLICATIONS
`
`U.S. PATENT DOCUMENTS
`4,931,950 A
`6/1990 Isle et al. ................... 364,513
`5,050,161 A
`9, 1991 Golestani ....
`... 370/60
`5,119,474 A
`6, 1992 Beitel et al. .
`... 395,154
`5,136,655 A * 8, 1992 Bronson .....
`... TO4,270
`5,274,758. A 12/1993 Beitel et al. .
`... 395,154
`5,309,562 A
`5/1994 Li .............................. 395/200
`5,313,454 A
`5, 1994 Bustini et al.
`37Of 13
`5,341,474 A
`8, 1994 Gelman et al. .
`... 395/200
`5,414,455 A
`5/1995 Hooper et al. ................. 348.7
`5.434,848 A
`7, 1995 Chimento, Jr. et al. ....... 37Of 17
`5,455,910 A 10/1995 Johnson et al. ............. 395/650
`5,481,542 A
`1/1996 Logston et al. ............ 700,
`5.490,252 A
`2, 1996 Macera et al. ......... 395.200.01
`5,504,744. A
`4, 1996 Adams et al. ............. 37060.
`5,519,701 A
`5/1996 Colmant et al. ........... 370/60.1
`5,521,630 A
`5/1996 Chen et al. .................... 348.7
`
`Eric Ladd et al. Using HTML 4, XML, and Java 1.2 (Que: Dec.
`1998), p. 690.*
`
`(Continued)
`
`Primary Examiner Amir Alavi
`(74) Attorney, Agent, or Firm—Lee & Hayes, PLLC
`
`(57)
`
`ABSTRACT
`
`E. a networked client/server system, media data is streamed
`rom a server to a client. A user of the client can search the
`-
`media data to identify portions that satisfy certain search
`criteria, and/or store the media data locally at the client for
`Subsequent playback.
`
`47 Claims, 8 Drawing Sheets
`
`302
`
`308
`
`308
`
`312
`
`Send Search Failure
`Natification to Cie
`
`Generats Index for a Media
`Stream
`
`
`
`304
`
`Additional Media
`Streams
`
`Receive Search Ragusst with
`Search Criteria
`
`Compare Indexes) of Media
`Streams Corresponding to
`Search Request to Search
`Citeia
`
`
`
`310
`
`Match entified
`2
`
`
`
`Identify Temparal
`Location in Media
`Streams Corresponding
`to Match
`
`3.14.
`
`8tream Data to Clent
`Starting at Match Location
`
`36
`
`IPR2020-00686
`Apple EX1028 Page 1
`
`

`

`US 7,149.359 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`7, 1998 Moran et al. ............... 345,328
`5,786,814 A
`8, 1998 Goldhaber et al. ........... 705/14
`5,794.210 A
`8, 1998 Orsolini et al. ...
`707/104
`5,794,249 A
`8/1998 Hekmatpour ..
`... 706.11
`5,799,292 A
`9, 1998 Miller et al. ...
`345,302
`5,801,685 A
`9/1998 Kinney et al. ...
`... 348/15
`5,808,662 A
`5,818,510 A 10/1998 Cobbley et al. ..
`... 348/7
`5,819,286 A * 10/1998 Yang et al. .......
`707/1
`5,822,537 A 10/1998 Katseff et al. ......... 395/200.61
`5,828,848. A
`10/1998 MacCormacket al. 395/200.77
`5,835,495 A 1 1/1998 Ferriere .........
`370/465
`5,835,667 A 1 1/1998 Wactlar et al. ................ 386/96
`5,838,906 A 11/1998 Doyle et al. ...
`395.200.32
`5,859,641 A
`1/1999 Cave ..........
`... 345.348
`5,864,682 A
`1/1999 Porter et al. ...
`395/2OO.77
`5,870,755 A
`2, 1999 Stevens et al. ............. TO7 104
`5,873,735 A
`2f1999 Yamada et al. ...
`... 434/316
`5,892.506 A
`4, 1999 Hermanson ....
`... 345,302
`5,903,673 A
`5/1999 Wang et al. ...
`... 382,236
`5,918,002 A
`6, 1999 Klemets et al. ........ 395,182.16
`5,930,493 A * 7/1999 Ottesen et al. ................ 725/92
`5,930,787 A
`7, 1999 Minakuchi et al. ............ 707/4
`5,953,506 A
`9, 1999 Kalra et al. ............ 395.200.61
`5,956,716 A
`9, 1999 Kenner et al. ................ 707/10
`5,973,679 A 10, 1999 Abbott et al. ..... ... T15,500.1
`5,983,236 A 11/1999 Yager et al. ...... ... 707/104.1
`5,995,941 A 11/1999 Maquire et al. .............. 70.5/10
`5.999,979 A 12/1999 Vellanki et al. ...... ... 709,232
`6,006.241 A 12/1999 Purnaveja et al. .......... 707/512
`6,014,706 A
`1/2000 Cannon et al. ............. 709/231
`6,023,731 A
`2, 2000 Chawla .........
`709,231
`6,032,130 A 2/2000 Alloul et al. .....
`... 705/27
`6,035,341 A
`3/2000 Nunally et al. ............. 709/253
`6,041,345 A
`3, 2000 Levi et al. .....
`... 709,217
`6,049,823. A
`4/2000 Hwang ..........
`709,218
`6,064,794. A
`5, 2000 McLaren et al. .
`... 386.68
`6,118,450 A
`9/2000 Proehl et al. ............... 345,349
`6,118,817 A
`9/2000 Wang ......................... 375,240
`6,128,653 A 10/2000 Del Val et al. ............. 709/219
`6,133,920 A 10/2000 DeCarmo et al. ........... 345,354
`6,144,375 A 11/2000 Jain et al. ................... 345,302
`6,148,304. A 11/2000 De Vries et al. ..
`... 707/104
`6,154,771 A 1 1/2000 Rangan et al. .............. 709/217
`6,166,314 A 12/2000 Weinstock et al. ......... 84.483.1
`6,173,317 B1
`1/2001 Chaddha et al. ..... ... 709,219
`6,185,573 B1* 2/2001 Angelucci et al. ....... TO7 104.1
`6,204,840 B1
`3/2001 Petelycky et al. .......... 345,302
`6,215,910 B1
`4/2001 Chaddha ...........
`... 382/253
`6,230,172 B1
`5/2001 Purnaveja et al. .......... 707/512
`6,233,389 B1
`5/2001 Barton et al. ................. 386/46
`6,239,801 B1* 5/2001 Chiu et al. .....
`715,500.1
`6,243,708 B1* 6/2001 deVries et al. .............. 707/102
`6,377,995 B1 * 4/2002 Agraharam et al. ........ TO9,231
`
`6,414.686 B1* 7/2002 Protheroe et al. ........... 345/474
`6,487,564 B1 * 1 1/2002 Asai et al. ...
`T15,500.1
`6,490,580 B1* 12/2002 Dey et al....................... 7O7/4
`6,636,238 B1 * 10/2003 Amir et al. ................. 345,730
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`E.
`WO
`WO
`
`5, 1995
`O653884
`8, 1995
`O669587 A2
`10, 1995
`O676898
`12/1996
`O746,158
`0812112 A2 12/1997
`WO 94/O1964
`1, 1994
`WO98/37698
`8, 1998
`
`OTHER PUBLICATIONS
`Haitao Jiang et al., “Spatial and temporal content-based access to
`hypervideo databases.” The VLDB Journal (1998), No. 7, pp.
`226-238.*
`Stacie Hibino et al., “MMVIS: Design and Implementation of a
`Multimedia Visual Information Seeking Environment.” Proceedings
`of the Fourth ACM International Conference on Multimedia (1996),
`pp. 75-86.*
`Stacie Hibino et al., “A Visual Query Language for Identifying
`Temporal Trends in Video Data.” IEEE 1995 Proceedings, Interna
`tional Workshop on Multi-Media Database Management Systems,
`pp. 74-81.*
`H.J. Chen et al., “A Scalable Video-on-Demand Service for the
`Provision of VCR-Like Functions.” IEEE, May 15, 1995, pp. 65-72.
`Lynda Hardman et al., “Multimedia authoring paradigms. Author
`ing and Application of Hypermedia-Based User-Interfaces, IEE
`Colloquium, The Institution of Electrical Engineers 1995, pp. 871
`8.3.
`John David N. Dionisio and Alfonso F. Cardenas, “A Unified Data
`Model for Representing Multimedia, Timeline, and Simulation
`Data.” IEEE Transactions on Knowledge and Data Engineering,
`vol. 10, No. 5, Sep./Oct. 1998, pp. 746-767.
`Arons, Barry, “SpeechSkimmer: A System for Interactively Skim
`ming Recorded Speech'. ACM Transactions on Computer-Human,
`vol. 4, No. 1, pp. 3-38.
`Internet Reference "An Annotated Bibliography of Interactive
`Speech User Interfaces by Barry Arons' HTTP://barons.www.
`media.mit.edu/people?barons. Arons Annotated Bibliography.html,
`Date unknown.
`“GSM Full Rate Speech Transcoding.” ETSI/PT 12, Feb. 1992, pp.
`1-93.
`P. Vary et al., “Speech Codec for the European Mobile Radio
`System.” 1998 IEEE, pp. 227-230.
`Informedia Internet Reference, http://www.informedia.cs.cmu.
`edu, date unknown.
`Microsoft Corporation and RealNetworks, Inc., Advanced Stream
`ing Format (ASF) Specification, Feb. 26, 1998, Public Specification
`Version 1.0, 55 pages.
`
`* cited by examiner
`
`IPR2020-00686
`Apple EX1028 Page 2
`
`

`

`U.S. Patent
`
`Dec. 12, 2006
`
`Sheet 1 of 8
`
`US 7,149,359 B1
`
`
`
`
`
`
`
`
`
`IPR2020-00686
`Apple EX1028 Page 3
`
`

`

`U.S. Patent
`
`Dec. 12, 2006
`
`Sheet 2 of 8
`
`US 7,149,359 B1
`
`Sng
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`IPR2020-00686
`Apple EX1028 Page 4
`
`

`

`U.S. Patent
`
`Dec. 12, 2006
`
`Sheet 3 of 8
`
`US 7,149,359 B1
`
`Ser109 -
`250
`252
`
`
`
`254
`
`- - - - - -
`256
`258
`
`248
`
`
`
`Composite
`Media Stream
`
`Client 104
`lent TU4
`204
`
`Stream Saving
`Module
`
`272
`
`Demultiplex
`
`214
`
`212
`
`210
`
`208
`
`2O6
`
`Animation
`Decode
`230
`
`Animation
`Render
`242
`
`Audio
`Render
`236
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`IPR2020-00686
`Apple EX1028 Page 5
`
`

`

`U.S. Patent
`
`Dec. 12, 2006
`
`Sheet 4 of 8
`
`US 7,149,359 B1
`
`
`
`
`
`
`
`Generate Index for a Media
`Stream
`
`Additional Media
`Streams
`
`
`
`
`
`
`
`No
`
`Receive Search Request with
`Search Criteria
`
`Compare Index(es) of Media
`Streams Corresponding to
`Search Request to Search
`Criteria
`
`
`
`
`
`
`
`
`
`Yes
`
`302
`
`306
`
`308
`
`Match dentified
`
`No
`
`
`
`312
`
`Send Search Failure
`Notification to Client
`
`ldentify Temporal
`Location in Media
`Streams Corresponding
`to Match
`
`
`
`
`
`Stream Data to Client
`Starting at Match Location
`
`
`
`314
`
`316
`
`IPR2020-00686
`Apple EX1028 Page 6
`
`

`

`U.S. Patent
`
`Dec. 12, 2006
`
`Sheet S of 8
`
`US 7,149,359 B1
`
`322
`
`Receive Search Request with
`Search Criteria
`
`Compare Media Streams
`Corresponding to Search
`Request to Search Criteria
`
`Match dentified
`
`
`
`
`
`
`
`328
`
`Send Search Failure
`Notification to Client
`
`Identify Temporal
`Location in Media
`Streams Corresponding
`to Match
`
`Stream Data to Client
`Starting at Match Location
`
`332
`
`
`
`
`
`
`
`
`
`IPR2020-00686
`Apple EX1028 Page 7
`
`

`

`U.S. Patent
`U.S. Patent
`
`ceD
`
`002
`
`teeh
`
`00f0
`
`US 7,149,359 B1
`US 7,149,359 B1
`
`
`
`
`
`58.5€82>
`
`EmE:oooazfmfimum
`
`979
`
`n86
`S|||||«mm
`6N3
`68m
`9%E53mm3%v2
`
`\8
`
`3‘
`\‘k
`
`|PR2020-00686
`
`Apple EX1028 Page 8
`
`IPR2020-00686
`Apple EX1028 Page 8
`
`

`

`U.S. Patent
`U.S. Patent
`
`Dec. 12, 2006
`
`Sheet 7 of 8
`
`US 7,149,359 B1
`US 7,149,359 B1
`
`
`
`
`
`(L) 13Al2S
`
`Document
`
`
`
`ImageStream
`
`
`
`AudioStream
`
`Markup
`
`
`
`TextStream
`
`Ecu
`a:
`.b
`(.0
`C
`.9._.
`as
`.EC
`<
`
`|PR2020-00686
`
`Apple EX1028 Page 9
`
`IPR2020-00686
`Apple EX1028 Page 9
`
`

`

`U.S. Patent
`
`Dec. 12, 2006
`
`Sheet 8 of 8
`
`US 7,149,359 B1
`
`390
`
`392
`
`394
`
`396
`
`398
`
`
`
`Receive Markup Document
`Referencing Media Stream(s)
`
`Receive Media Stream(s)
`Referenced in Markup
`DOCument
`
`Store Media Stream(s) Locally
`
`Modify Markup Document to
`Reference Locally Stored Media
`Stream(s)
`
`Store Modified Markup
`Document Locally
`
`Package Locally Stored Media
`Stream(s) and Modified Markup
`Document Together
`(Optional)
`
`IPR2020-00686
`Apple EX1028 Page 10
`
`

`

`US 7,149,359 B1
`
`1.
`SEARCHING AND RECORDING MEDIA
`STREAMS
`
`TECHNICAL FIELD
`
`This invention relates to networked client/server systems
`and to searching and recording streaming media content in
`Such systems.
`
`BACKGROUND OF THE INVENTION
`
`10
`
`2
`The invention described below addresses these disadvan
`tages, providing for the searching and recording of stream
`ing media content.
`
`SUMMARY OF THE INVENTION
`
`In a networked client/server system, media content is
`streamed from the server to the client. A user of the client
`can search the media content to identify temporal locations
`that satisfy certain search criteria, and/or store the media
`content locally at the client for Subsequent playback.
`According to one aspect of the invention, indexes are
`maintained for each of different media streams that can be
`streamed to the client either individually or together for a
`multimedia presentation. The indexes store a correspon
`dence between content for a media stream and temporal
`locations of that media stream. In response to a user search
`request, search criteria is compared to the appropriate
`index(es) to identify whether the search criteria matches any
`data in the index(es).
`According to another aspect of the invention, in response
`to a user search request the search criteria from the search
`request is compared directly to the media stream data rather
`than to an index. This comparison is made to identify
`whether the search criteria matches any of the media stream
`data.
`According to another aspect of the invention, if data
`matching the search criteria is found (either in an index or
`the media stream data), then the media server “seeks” to a
`temporal location of the media stream identified by the
`matching data. The server then proceeds to stream the media
`content to the client beginning at that temporal location.
`According to another aspect of the invention, a search
`request and corresponding search criteria are compared to
`multiple media streams (either directly or indirectly via
`associated indexes). Thus, a single search request can be
`used to search through all of the media streams of a
`multimedia presentation.
`According to another aspect of the invention, the multiple
`media streams of a multimedia presentation are streamed
`from the server to the client and stored locally by the client.
`A markup document, referencing the multiple media streams
`also stored locally at the client, is generated and stored at the
`client. Thus, a user can play back the locally stored multi
`media presentation at a later time when not coupled to a
`network and thus not able to receive streaming media from
`the server.
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`Multimedia streaming the continuous delivery of syn
`chronized media data like Video, audio, text, and anima
`tion is a critical link in the digital multimedia revolution.
`Today, streaming media is primarily about video and audio,
`but a richer, broader digital media era is emerging with a
`profound and growing impact on the Internet and digital
`broadcasting.
`Synchronized media means multiple media objects that
`share a common timeline. Video and audio are examples of
`synchronized media—each is a separate data stream with its
`own data structure, but the two data streams are played back
`in Synchronization with each other. Virtually any media type
`can have a timeline. For example, an image object can
`change like an animated .gif file, text can change and move,
`and animation and digital effects happen over time. This
`concept of synchronizing multiple media types is gaining
`greater meaning and currency with the emergence of more
`Sophisticated media composition frameworks implied by
`MPEG-4, Dynamic HTML, and other media playback envi
`rOnmentS.
`The term “streaming is used to indicate that the data
`representing the various media types is provided over a
`network to a client computer on a real-time, as-needed basis,
`rather than being pre-delivered in its entirety before play
`back. Thus, the client computer renders streaming data as it
`is received from a network server, rather than waiting for an
`entire “file to be delivered.
`Streaming multimedia content enables a variety of infor
`mational content that was not previously available over the
`Internet or other computer networks. Live content is one
`significant example of Such content. Using streaming mul
`timedia, audio, video, or audio/visual coverage of notewor
`thy events can be broadcast over the Internet as the events
`unfold. Similarly, television and radio stations can transmit
`their live content over the Internet.
`However, one current problem with streaming multimedia
`content is that users are typically limited to accessing the
`multimedia content via common 'shuttle controls” on a
`multimedia player. Such as a play button, fast forward
`button, pause button, etc. Given that large amounts of data
`can be stored as multimedia content (e.g., individual pre
`sentations lasting for hours). Such controls make it difficult
`for a user to locate the portions of the multimedia content
`that are of most interest to him or her.
`An additional problem with streaming multimedia content
`is that the user must typically be connected to the same
`network as the server (e.g., the Internet) in order to receive
`the streaming multimedia content. If this connection is not
`maintained then the streaming of the multimedia content
`stops. This "continuous connection' limitation can be
`troublesome for many individuals, such as those using
`portable computers in locations that may not always have
`access to the appropriate network, or individuals who do not
`want to tie up a telephone line for their network connection
`while playing back the multimedia content.
`
`50
`
`55
`
`60
`
`65
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention is illustrated by way of example
`and not limitation in the figures of the accompanying
`drawings. The same numbers are used throughout the figures
`to reference like components and/or features.
`FIG. 1 shows a client/server network system and envi
`ronment in accordance with the invention.
`FIG. 2 shows a general example of a computer that can be
`used in accordance with the invention.
`FIG. 3 illustrates an exemplary client-server relationship
`for streaming data.
`FIG. 4 is a flowchart illustrating an exemplary process for
`searching media streams in accordance with one implemen
`tation of the invention.
`FIG. 5 is a flowchart illustrating another exemplary pro
`cess for searching media streams in accordance with another
`implementation of the invention.
`
`IPR2020-00686
`Apple EX1028 Page 11
`
`

`

`US 7,149,359 B1
`
`3
`FIGS. 6 and 7 are block diagrams illustrating the local
`storage of a multimedia presentation in accordance with one
`implementation of the invention.
`FIG. 8 is a flowchart illustrating an exemplary process for
`recording a multimedia presentation in accordance with one
`implementation of the invention.
`
`DETAILED DESCRIPTION
`
`4
`presentation using a particular presentation medium. The
`individual media streams can be rendered to produce a
`plurality of different types of user-perceivable media,
`including synchronized audio or sound, video graphics or
`motion pictures, animation, textual content, command Script
`sequences, or other media types that convey time-varying
`information or content in a way that can be sensed and
`perceived by a human. The individual media streams have
`their own timelines, which are synchronized with each other
`so that the media streams can be rendered simultaneously for
`a coordinated multimedia presentation. These individual
`media streams can be delivered to the client computer as
`individual streams from one or more servers, as a composite
`media stream(s) from one or more servers, or a combination
`thereof.
`In this discussion, the term “composite media stream”
`describes synchronized streaming data that represents a
`segment of multimedia content. The composite media
`stream has a timeline that establishes the speed at which the
`content is rendered. The composite media stream can be
`rendered to produce a plurality of different types of user
`perceivable media, such as Synchronized audio or sound,
`Video graphics or motion pictures, animation, textual con
`tent, command Script sequences, etc. A composite media
`stream includes a plurality of individual media streams
`representing the multimedia content.
`There are various standards for streaming media content
`and composite media streams. “Advanced Streaming For
`mat’ (ASF) is an example of such a standard, including both
`accepted versions of the standard and proposed standards for
`future adoption. ASF specifies the way in which multimedia
`content is stored, streamed, and presented by the tools,
`servers, and clients of various multimedia vendors. ASF
`provides benefits such as local and network playback, exten
`sible media types, component download, Scalable media
`types, prioritization of streams, multiple language Support,
`environment independence, rich inter-stream relationships,
`and expandability. Further details about ASF are available
`from Microsoft Corporation of Redmond, Wash.
`Regardless of the streaming format used, an individual
`data stream contains a sequence of digital data units that are
`rendered individually, in sequence, to produce an image,
`Sound, or some other stimuli that is perceived by a human to
`be continuously varying. For example, an audio data stream
`comprises a sequence of Sample values that are converted to
`a pitch and Volume to produce continuously varying Sound.
`A video data stream comprises a sequence of digitally
`specified graphics frames that are rendered in sequence to
`produce a moving picture. An animation stream comprises a
`sequence of graphical images that are rendered in sequence
`to produce a moving image. An image stream comprises a
`sequence of graphical images that are rendered to produce a
`changing image over time. A text stream is a sequence of
`symbols and/or alphanumeric characters that are rendered to
`produce different symbol/character combinations over time
`(e.g., in the form of words).
`For a composite media stream, the individual data streams
`are typically interleaved in a single sequence of data packets.
`Various types of data compression might be used within a
`particular data format to reduce communications bandwidth
`requirements.
`The sequential data units (such as audio sample values,
`Video frames, groups of characters, graphical images, etc.)
`of the individual streams are associated with both delivery
`times and presentation times, relative to an arbitrary start
`time. The delivery time of a data unit indicates when the data
`unit should be delivered to a rendering client. The presen
`
`10
`
`25
`
`30
`
`35
`
`General Network Structure
`FIG. 1 shows a client/server network system and envi
`ronment in accordance with the invention. Generally, the
`system includes one or more (m) network multimedia server
`computers 100, one or more (Z) index server computers 102,
`15
`and one or more (n) network client computers 104. The
`computers communicate with each other over a data com
`munications network, which in FIG. 1 includes a public
`network 106 such as the Internet. The data communications
`network might also include local-area networks and/or pri
`vate wide-area networks. Server computers 100 and client
`computers 104 communicate with one another via any of a
`wide variety of known protocols, such as the Hypertext
`Transfer Protocol (HTTP).
`Multimedia servers 100 have access to streaming media
`content in the form of different media streams. These media
`streams can be individual media streams (e.g., audio, video,
`graphical, text, etc.), or alternatively composite media
`streams including multiple Such individual streams. Some
`media streams might be stored as files 108 in a database or
`other file storage system, while other media streams 110
`might be supplied to the server on a “live' basis from other
`data source components through dedicated communications
`channels or through the Internet itself.
`The media streams received from servers 100 are ren
`dered at the client computers 104 as a multimedia presen
`tation, which can include media streams from one or more
`of the servers 100. These different media streams can
`include one or more of the same or different types of media
`streams. For example, a multimedia presentation may
`40
`include two video streams, one audio stream, and one stream
`of graphical images.
`A user interface (UI) at the client computer 104 allows
`users to control the playback of the multimedia presentation,
`Such as selecting which of multiple presentations to play
`back, controlling pausing of the playback, etc. The UI at
`client 104 further allows a user to input search criteria for
`searching one or more of the individual media streams
`available from a server 100, and to save the media streams
`of a multimedia presentation for Subsequent playback when
`not coupled to network 106.
`Index servers 102 optionally maintain indexes for the
`streaming media data available from servers 100. These
`indexes provide a correspondence between elements or
`objects of the media data streams and temporal locations of
`the media data streams. These indexes can be used for
`searching the media data streams, as discussed in more detail
`below. Alternatively, the indexes may be maintained at the
`media servers 100.
`Streaming Media
`In this discussion, streaming media refers to one or more
`individual media streams being transferred over a network to
`a client computer on an as-needed basis rather than being
`pre-delivered in their entirety before playback. Each of the
`individual media streams corresponds to and represents a
`different media type and each of the media streams can be
`rendered by a network client to produce a user-perceivable
`
`50
`
`45
`
`55
`
`60
`
`65
`
`IPR2020-00686
`Apple EX1028 Page 12
`
`

`

`US 7,149,359 B1
`
`5
`
`5
`tation time indicates when the value should be actually
`rendered. Normally, the delivery time of a data unit precedes
`the presentation time.
`The presentation times determine the actual speed of
`playback. For data streams representing actual events or
`performances, the presentation times correspond to the rela
`tive times at which the data samples were actually recorded.
`The presentation times of the various different individual
`data streams are consistent with each other so that the
`streams remain coordinated and synchronized during play
`back.
`Exemplary Computer Environment
`In the discussion below, the invention will be described in
`the general context of computer-executable instructions,
`15
`Such as program modules, being executed by one or more
`conventional personal computers. Generally, program mod
`ules include routines, programs, objects, components, data
`structures, etc. that perform particular tasks or implement
`particular abstract data types. Moreover, those skilled in the
`art will appreciate that the invention may be practiced with
`other computer system configurations, including hand-held
`devices, multiprocessor systems, microprocessor-based or
`programmable consumer electronics, network PCs, mini
`computers, mainframe computers, and the like. In a distrib
`uted computer environment, program modules may be
`located in both local and remote memory storage devices.
`Alternatively, the invention could be implemented in
`hardware or a combination of hardware, software, and/or
`firmware. For example, one or more application specific
`integrated circuits (ASICs) could be programmed to carry
`out the invention.
`FIG. 2 shows a general example of a computer 142 that
`can be used in accordance with the invention. Computer 142
`is shown as an example of a computer that can perform the
`functions of any of server computers 100 or 102, or client
`computers 104 of FIG. 1.
`Computer 142 includes one or more processors or pro
`cessing units 144, a system memory 146, and a bus 148 that
`couples various system components including the system
`memory 146 to processors 144.
`The bus 148 represents one or more of any of several
`types of bus structures, including a memory bus or memory
`controller, a peripheral bus, an accelerated graphics port, and
`a processor or local bus using any of a variety of bus
`architectures. The system memory includes read only
`memory (ROM) 150 and random access memory (RAM)
`152. A basic input/output system (BIOS) 154, containing the
`basic routines that help to transfer information between
`elements within computer 142. Such as during start-up, is
`stored in ROM 150. Computer 142 further includes a hard
`disk drive 156 for reading from and writing to a hard disk,
`not shown, a magnetic disk drive 158 for reading from and
`writing to a removable magnetic disk 160, and an optical
`disk drive 162 for reading from or writing to a removable
`optical disk 164 such as a CD ROM or other optical media.
`The hard disk drive 156, magnetic disk drive 158, and
`optical disk drive 162 are connected to the system bus 148
`by an SCSI interface 166 or some other appropriate inter
`face. The drives and their associated computer-readable
`media provide nonvolatile storage of computer readable
`instructions, data structures, program modules and other
`data for computer 142. Although the exemplary environment
`described herein employs a hard disk, a removable magnetic
`disk 160 and a removable optical disk 164, it should be
`appreciated by those skilled in the art that other types of
`computer readable media which can store data that is
`
`25
`
`30
`
`35
`
`40
`
`45
`
`6
`accessible by a computer. Such as magnetic cassettes, flash
`memory cards, digital video disks, random access memories
`(RAMs) read only memories (ROM), and the like, may also
`be used in the exemplary operating environment.
`A number of program modules may be stored on the hard
`disk, magnetic disk 160, optical disk 164, ROM 150, or
`RAM 152, including an operating system 170, one or more
`application programs 172, other program modules 174, and
`program data 176. A user may enter commands and infor
`mation into computer 142 through input devices such as
`keyboard 178 and pointing device 180. Other input devices
`(not shown) may include a microphone, joystick, game pad,
`satellite dish, scanner, or the like. These and other input
`devices are connected to the processing unit 144 through an
`interface 182 that is coupled to the system bus. A monitor
`184 or other type of display device is also connected to the
`system bus 148 via an interface, such as a video adapter 186.
`In addition to the monitor, personal computers typically
`include other peripheral output devices (not shown) such as
`speakers and printers.
`Computer 142 operates in a networked environment using
`logical connections to one or more remote computers, such
`as a remote computer 188. The remote computer 188 may be
`another personal computer, a server, a router, a network PC,
`a peer device or other common network node, and typically
`includes many or all of the elements described above
`relative to computer 142, although only a memory storage
`device 190 has been illustrated in FIG. 2. The logical
`connections depicted in FIG. 2 include a local area network
`(LAN) 192 and a wide area network (WAN) 194. Such
`networking environments are commonplace in offices, enter
`prise-wide computer networks, intranets, and the Internet. In
`the described embodiment of the invention, remote com
`puter 188 executes an Internet Web browser program such as
`the “Internet Explorer Web browser manufactured and
`distributed by Microsoft Corporation of Redmond, Wash.
`When used in a LAN networking environment, computer
`142 is connected to the local network 192 through a network
`interface or adapter 196. When used in a WAN networking
`environment, computer 142 typically includes a modem 198
`or other means for establishing communications over the
`wide area network 194, such as the Internet. The modem
`198, which may be internal or external, is connected to the
`system bus 148 via a serial port interface 168. In a net
`worked environment, program modules depicted relative to
`the personal computer 142, or portions thereof, may be
`stored in the remote memory storage device. It will be
`appreciated that the network connections shown are exem
`plary and other means of establishing a communications link
`between the computers may be used.
`Generally, the data processors of computer 142 are pro
`grammed by means of instructions stored at different times
`in the various computer-readable storage media of t