Ulllted States Patent
`
`[19]
`
`[11] Patent Number:
`
`5,956,729
`
`Goetz et al.
`
`[45] Date of Patent:
`
`Sep. 21, 1999
`
`US005956729A
`
`[54] MULTIMEDIA FILE, SUPPORTING
`
`AND METHOD FOR FORMING SAME
`
`[75]
`
`Inventors: Tom Goetz, Attleboro; Manickam R.
`Sridhar, Holliston; Mllkesh Prasad,
`Chestnut Hill, all of Mass.
`
`[73] Assignee: Motorola, Inc., Schaumburg, Ill.
`
`[21] App}, No; 08/711,696
`.
`Ffled:
`
`Sep.6,1996
`
`[22]
`
`5,737,595
`5,768,527
`5,774,583
`
`......................... .. 707/100
`4/1998 Cohen et al.
`
`395/200.61
`6/1998 Zhu et al.
`......................... .. 382/190
`6/1998 Sasaki et al.
`OTHER PUBLICATIONS
`
`RealMedia Overview, “The First Open Multimedia Stream-
`ing
`Platform,”
`http://www.realuadio.com/
`or
`e—mail
`radinfo@prognet.com; copyright © Progressive Networks,
`1996,
`Paper,”
`White
`“Technical
`RealMedia,
`http://wvvw.realuadio.com/ or e—mail
`radinfo @prognet-
`.com; copyright © Progressive Networks, 1996.
`
`6
`
`Int‘ CL‘
`[51]
`“““ G06F17/30
`
`[52] U.S. CI.
`............. ..
`. 707/104, 707/102; 707/103
`[58] Field of Search ................................... .. 707/102-104,
`707/200’ 1
`
`Primary Examiner—Maria N. Von Buhr
`Attorney, Agent, or Firm—Peter M. Dicliiara; Jeffrey T.
`Klayman
`_
`[37]
`
`ABSTRACT
`
`Amultimedia file and method for forming the same organize
`instances of inultiinedia information according to media
`information type (e.g., audio, video, MIDI, etc.), encoding
`format, media subtype, and encoding rate. Several instances
`of the Same media information t
`6 are included each of
`yp
`.
`.
`.
`.
`’
`.
`such instances having a different encoding format, media
`subtype, and/or encoding rate. A presentation application
`utilizes the Subject multimedia file to identify, Select, and
`present specific instances of the multimedia information,
`permitting the presentation application to customize a mul-
`timedia presentation based on, among other things, the rate
`of the connection from the presentation application to the
`presentation consumer and the decoding capabilities of the
`~
`Presemanon Consumer‘
`
`31 Claims, 8 Drawing Sheets
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,333,299
`.......................... .. 395/650
`7/1994 Koval et al.
`5,339,413
`..
`395/550
`8/1994 Kovaletal.
`5,487,167
`395/650
`11,1996 Dinallo et al.
`5,504,892
`707/103
`N
`4/1996 Atsatt at al.
`5,574,898
`.......................... .. 707/1
`11/1996 LeBlang et al.
`5,574,905
`12/1996 de Carmo ................................. .. 707/1
`5,586,255
`12/1996 Tanaka et al.
`395/200.53
`5,608,859
`3/1997 Taglwhi ~~~~~~~ ~~
`345/302
`55449756
`7/1997 C03’ 6‘ a1~
`707/204
`g1ereII11bergtet1a]'
`,
`,
`,
`0118. 116 C 8.
`,
`.
`.
`.
`5,696,500 12/1997 Diem ........... ..
`340/825.44
`5,701,465
`12/1997 Baugher et al.
`.... .. 707/10
`5,732,256
`3/1998 Smith ........................................ .. 707/1
`
`
`
`..
`
`FILE HEADER
`PREAMBLE
`
`MEDIA INSIANCE
`DESCRIPTOR
`
`22
`
`ZIO
`
`0
`
`16
`FILE SIGNATURE
`HEADER SIZE
`VERSION won VERSION MINOR
`uuuasn or MEDIA INSTANCES
`
`- RESERVED
`
`W— RESERVED
`
`
`214
`
`31
`
`
`
`ZII
`272
`215
`
`216
`
`217
`
`
`0
`
`
`
`
`
`
`
`M's’\
`
`16
`MEDIA BLOCK OFFSET
`
`MEDIA IYPE
`
`ENCODING TYPE
`
`SUBTYPE
`
`RATE
`
`22!
`
`31
`
`ZZZ
`
`
`
`
`
`
`RPX Exhibit 1041
`RPX Exhibit 1041
`RPX v. DAE
`RPX V. DAE
`
`

`
`U.S. Patent
`
`Sep.21,1999
`
`Sheet 1 of8
`
`5,956,729
`
`I00
`
`('
`
`FILE
`HEADER
`
`110
`
`
`FIG 1
`
`FILE
`BODY
`
`
`
`'2”
`
`
`
`
`
`5;
`
`FILE HEADER
`PREAMBLE
`
`MEDIA INSTANCE
`DESCRIPTOR
`
`270
`
`22
`
`22
`
`22
`
`210
`
`0
`
`
`
`
`
`16
`
`FILE SIGNATURE
`
`HEADER SIZE
`
`
`
`31
`
`NUMBER OF MEDIA INSTANCES
`VERSION MINOR
`VERSION MAJOR
`_ RESERVED
`3'3- REsERvED
`
`214
`
`F I G.2B
`
`277
`
`272
`215
`
`216
`217
`
`0
`
`16
`
`MEDIA BLOCK OFFSET
`
`MEDIA TYPE
`
`ENCODING TYPE
`
`SUBTYPE
`
`RAIE
`
`FIG.2C
`
`220
`
`31
`
`22:
`
`222
`
`223
`
`224
`
`225
`
`
`
`

`
`U.S. Patent
`
`Sep.21,1999
`
`Sheet 2 of8
`
`5,956,729
`
`W :--I
`
`(‘I20
`
`/-310
`
`MEDIA
`
`$
`
`o13Eg%Rv
`
`.320
`
`330
`
`F I C.-4A
`
`/'40”
`
`
`
`no
`
`420
`
`o
`
`31
`
`4"
`
`PACKET coum
`
`PRESENIATION LENGIH
`
`RESERVED
`
`412
`413
`
`
`
`(
`FIG.4B
`
`410
`
`8
`
`16
`
`31
`
`0
`
`
`
`
`427
`4??
`
`
`4,,
`F I 0.46‘
`425
`
`

`
`U.S. Patent
`
`Sep.21,1999
`
`Sheet 3 of8
`
`5,956,729
`
`441
`
`442
`
`443
`
`444
`
`8
`
`IMPORTANCE
`CHANNEL
`PACKET LENGTH
`
`15
`
`SEGMENT
`
`13
`
`TOTAL SEGMENTS
`RESERVED
`
`445
`
`
`
`SEQUENCE NUMBER
`447
`j 443
`
`—1_ 450
`
`
`
`W
`
`FI(}.4E
`
`
`
`3
`
`57}
`
`
`
`574
`
`511
`
`512
`
`
`16
`PACKET COUNT
`
`MAX FRAME SIZE
`
`PRESENTATION LENGTH
`
`i FRAME COUNT
`
`O
`
`
`
`
`611
`
`
`16
`
`PACKET COUNT
`
`O
`
`
`
`612
`
`MIDI FORMAT
`
`
` 614
`
`
`PRESENTATION LENGTH
`TRACK coum —
`DIVISOR
`
`3
`
`573
`
` 616
`
`
`
`

`
`U.S. Patent
`
`Sep.21,1999
`
`Sheet 4 of 8
`
`5,956,729
`
`8H7
`
`CAMERA
`
`805
`
`
`815
`
`MICROPHONE
`
`825
`
`835
`
`FIG.8
`
`
`SERVER
`
`CLIENT
`
`FIG.9
`
`

`
`U.S. Patent
`
`Sep.21,1999
`
`Sheet 5 of8
`
`5,956,729
`
`CLIENT
`
`SERVER
`
`WEB
`BROWSER
`APPLICATION
`
`MULTIMEDIA
`CLIENT
`APPLICATION
`
`WEB
`SERVER
`APPLICATION
`
`WEB PAGES
`
`MULTIMEDIA
`FILE
`
`910
`
`920
`
`

`
`U.S. Patent
`
`Sep.21,1999
`
`Sheet 6 of8
`
`5,956,729
`
`CLIENT
`
`SERVER
`
`0
`
`SEND MEDIA
`
`REQUEST
`MESSAGE
`
`SEND MESSAGE
`
`SPECIFYING
`
`RATE
`
`SEND "G0"
`MESSAGE
`
`SEND RESPONSE
`
`0
`
`SEND MEDIA
`RESPONSE
`MESSAGE
`
`SEND MESSAGE
`
`FOR CALCULATING
`ROUND-TRIP
`
`DELAY
`
`SEND CONFIG.
`MESSAGE
`
`SEND "G0"
`
`MESSAGE
`
`FIGJ1
`
`

`
`U.S. Patent
`
`Sep.21,1999
`
`Sheet 7 of 8
`
`5,956,729
`
`099 GO TO SLEEP
`
`RETRANSMIT
`
`?
`
`GO TO SLEEP
`
`
`
`MORE
`SLEEP TIME
`
`I265
`
`FIGJ2
`
`

`
`U.S. Patent
`
`Sep.21,1999
`
`Sheet 8 of8
`
`5,956,729
`
`START
`
`A300
`
`
`
`
`
`
`
`MODEM
`RELATED PROBLEM
`
`AND THROUGHPUT
`
`DECREASING ?
`
`
`
`
`
`'35”
`
`RANDOM
`ERRORS
`?
`YES
`
`NETWOR
`PROELEM
`NO
`
`I330
`
`
`
`SEND PACKETS
`
`L399
`
`FIGJ3
`
`
`DON'T THROTTLE
`
`
`THROTTLE novm
`maomz UP
`(DROP PACKET IF
`(RETRANSMIT IF
`IMPORTANCE > 4)
`IMPORTANCE < 3)
`
`
`
`
`
`THROTTLE DOWN
`(DROP PACKET IF
`IMPORTANCE = 5)
`
`
`
`

`
`5,956,729
`
`1
`MULTIMEDIA FILE, SUPPORTING
`MULTIPLE INSTANCES OF MEDIA TYPES,
`AND METHOD FOR FORMING SAME
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is related to the following U.S. patent
`applications, all of which are assigned to the assignee of this
`application and all of which are incorporated by reference
`herein:
`
`Dcvicc, System And Method Of Rcal-Timc Multimcdia
`Streaming (Ser. No. 08/636,417, to Qin-Fan Zhu, Man-
`ickam R. Sridhar, and M. Vedat Eyuboglu, filed Apr.
`23, 1996, now U.S. Pat. No. 5,768,527 (issued Jun. 16,
`1998)
`Improved Video Encoding System And Method (Ser. No.
`08/711,702,
`to Manickam R. Sridhar and Feng Chi
`Wang, filed on Sep. 6, 1996, pending and
`System, Device, And Method For Streaming A Multime-
`dia File Method (Ser. No. 08/711,701, to Manickam R.
`Sridhar, Tom Goetz and Mukesh Prasad, filed on Sep.
`6, 1996 herewith, pending.
`BACKGROUND
`
`1. Field of the Invention
`
`The invention generally relates to real-time multimedia
`applications and, more particularly,
`to the streaming of
`real-time multimedia information over a communication
`network.
`2. Discussion of Related Art
`
`Generally speaking, multimedia applications present
`rclatcd media information, such as video, audio, music, ctc.,
`on a presentation device, such as a computer having a
`display and sound system. Some multimedia applications
`are highly interactive, where as other applications are far less
`interactive. For example, a game is a highly interactive
`application in which the application must respond to many
`user inputs such as keyboard commands and joystick
`movements, whereas viewing a video clip is less interactive
`and may only involve start and stop commands. Moreover,
`multimedia applications may be directed to standalone
`single computer contexts, or they may be directed to
`distributed, network-based contexts.
`following the
`At a very high lcvcl of abstraction,
`producer-consumer paradigm, any multimedia application
`involves producing and consuming the related multimedia
`information. The above examples of highly interactive and
`less interactive and standalone and network-based applica-
`tions differ
`in the manner in which the information is
`produced and consumed and the complexity in controlling
`the production and consumption.
`For example, in PCs and other standalone contexts, the
`information need only be read olf a local CD Rom or the
`like, and thus, producing the information to the consumer
`involves relatively predictable characteristics and requires
`rclativcly simplc control logic. In nctwork-bascd contcxts,
`on the other hand, the information must be produced over
`the network and is thus subject to the unpredictable char-
`acteristics and intricacies of the network: data may be lost,
`performance may vary over time, and so on. Consequently,
`the control logic may need to be relatively complicated.
`In either the standalone or network-based context, con-
`suming the information involves presenting the related
`information to the corresponding presentation components
`in a controlled manner and in real time. For example, to
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`provide intelligible audio-video clips, the video data must be
`provided to a video driver and audio data must be provided
`to a sound card driver within specified timing tolerances to
`maintain intra- and inter-stream synchronism. Intra-stream
`synchronism means that a given stream, such as audio, is
`presented in synchronism within specified time
`relationships,
`in short, that thc strcam itself is cohcrcnt.
`Inter-stream synchronism means that multiple related
`streams are presented in synchronism with respect to each
`other. Concerning intra-stream synchronism, all streams, for
`the most part, should present data in order. However, users
`are more forgiving if some streams, such as video, leave out
`certain portions of the data than they are of other streams,
`such as audio, doing the same. Avideo stream with missing
`data may appear a little choppy, but an audio stream with
`missing data may be completely unintelligible. Concerning
`inter-stream synchronism, poor control will likely result in
`poor “lip synch,” making the presentation appear and sound
`like a poorly dubbed movie.
`In the network-based context, one simple model of pro-
`ducing the information involves the consuming entity to
`request the downloading of the multimedia information for
`an entire presentation from a server, storing the multimedia
`information. Once downloaded,
`the client may then
`consume, or present, the information. Although relatively
`simple to implement, this model has the disadvantage of
`requiring the user to wait for the downloading to complete
`before the presentation can begin. This delay can be con-
`siderable and is especially annoying when a user finds that
`he or she is only interested in a small portion of the overall
`presentation.
`A more sophisticated model of producing information
`involves a server at one network site “streaming” the mul-
`timedia information over the network to a client at another
`
`site. The client begins to present the information as it arrives,
`rather than waiting for the entire data set to arrive before
`beginning presentation. This benefit of reduced delay is at
`the expense of increased complexity. Without the proper
`control, data overflow and underflow may occur, seriously
`degrading the quality of the presentation.
`Many modem multimedia applications involve the trans-
`fer of a large amount of information, placing a considerable
`load on the resources of the network, server, and client. The
`use of network-based multimedia applications appears to be
`growing. As computers become more powerful and more
`people access network-based multimedia applications, there
`will be an increased demand for longer, more complicated,
`more flexible multimedia applications, thereby placing even
`larger loads and demands on the network, server, and client.
`The demand placed on servers by these ever-growing mul-
`timedia applications is particularly high, as individual serv-
`ers are called upon to support larger numbers of simulta-
`neous uses: it is not uncommon even today for an Internet
`server to handle thousands of simultaneous channels.
`Consequently, there is a need in the art for a device, system,
`and method that, among other things,
`can handle longer, more complicated presentations;
`utilize a network’s resources more efficiently; and
`utilize a server’s and client’s resources more efficiently.
`SUMMARY
`
`the invention involves a new file format for
`In short,
`organizing related multimedia information and a system and
`device for, and method of, using the new file format. The
`invention eases the management and control of multimedia
`presentations, having various media streams, each of a
`
`

`
`5,956,729
`
`3
`specific type, each specific type further classified by encod-
`ing type, subtype, and encoding rate. Thus, with the
`invention, an application may support several instances of a
`particular media type, with each instance having different
`characteristics. For example, the application may support
`multiple audio streams, with each stream in a different
`language, from which a user may select. Analogously, with
`the invention, the application may choose a given instance
`of a media stream based on the network’s characteristics; for
`example, the application may choose an audio subtype that
`is encoded for a transmission rate that matches the network’s
`characteristics. The invention reduces a server’s memory
`and processing requirements,
`thus allowing a server to
`simultaneously service more requests and support more
`channels. And, the invention dynamically adapts the media’s
`streaming rate to use the netWorl<’s resources more efli—
`ciently while minimizing the effects of the adaptation on the
`quality of the presentation.
`The invention includes a multimedia file for organizing at
`least one type of media information on a computer-readable
`medium, such as a CD Rom, hard disk, or the like. The
`multimedia file is capable of storing and identifying multiple
`instances of the at least one media type. For example, the
`media information types may include audio, video, MIDI,
`etc., and the multiple instances may correspond to a French
`instance of audio, an English instance, and a Chinese
`instance. In this fashion, a presentation application, using
`the appropriate logic, may select and present any of the
`multiple instances of the media type.
`One embodiment organizes the file as a file body for
`storing a plurality of media blocks, each containing infor-
`mation for a corresponding instance, and as a file header
`having information for referencing the contents of the file
`body. The header may include media instance descriptors,
`each including information for describing the media block
`and including information for locating the data in the body.
`Some attributes of a given instance included in the descrip-
`tor include a media type, e.g., audio, an encoding type, e.g.,
`MPEG, a subtype, e.g., English, and a streaming, or
`decoding, rate.
`One embodiment organizes the data in the body in pack-
`etized form, the form corresponding to a predefined network
`protocol, such as the UDP interface of the TCP/IP protocol
`suite. The packets may or may not be on a one-to-one
`correspondence with the presentation units, eventually pro-
`cessed by the presentation application. Thus a single packet
`may have several audio blocks merged into it, or a single
`video frame may be divided into several packets.
`The packet descriptors, used to describe and locate the
`packets, as well as the packets themselves may include
`importance information indicating the importance of a given
`packet to the perceived quality of the eventual presentation.
`Some packets are critical, whereas other may be dropped
`without severely degrading the quality of the presentation.
`Keeping information in prepacketized form, requires less
`resources by servers and the like, using files organized
`according to the format. Among other things,
`the format
`alleviates servers from having to keep huge packet windows.
`To form a file according to the format, one embodiment
`forms a file body, containing at least two instances of the at
`least one type of media information, and forms a file header
`identifying each instance so that a presentation application
`may select and present any of the media types and any of the
`instances thereof.
`
`To form the body, media information is encoded into an
`encoded media stream, which is then packetized, and then
`formed into a media block.
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`The encoding operation n1ay include encoding the media
`stream for a pre-determined data rate, and the packetizing
`operation may include the step of recording an assigned
`importance value to each packet.
`Due to its structure, the file format may be easily edited
`to include or delete instances of media information.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`In the Drawing,
`FIG. 1 is a block diagram showing the novel multimedia
`file at a high level of abstraction;
`FIG. 2A is a block diagram showing the file header of the
`multimedia file;.
`FIG. 2B is a block diagram showing the format of a file
`header preamble;
`FIG. 2C is a block diagram showing the format of a media
`instance descriptor;
`FIG. 3 is a block diagram showing the format of a media
`block;
`FIG. 4A is a block diagram showing a generic media
`block directory format;
`FIG. 4B is a block diagram showing the format of a
`generic directory preamble;
`FIG. 4C is a block diagram showing the format of a
`generic packet descriptor;
`FIG. 4D is a block diagram showing the format of a
`generic media block body;
`FIG. 4E is a block diagram showing the format of a
`generic packet;
`FIG. 5 is a block diagram showing the format of an H.263
`media block;
`FIG. 6 is a block diagram showing the format of a MIDI
`media block preamble;
`FIG. 7 is a block diagram showing the format of a MIDI
`packet;
`FIG. 8 is a block diagram showing a system for creating
`a multimedia file;
`FIG. 9 is a block diagram showing a system embodying
`the invention in a client-server context;
`FIG. 10 is a block diagram showing the client and server
`components of the system;
`FIG. 11 is a flow diagram showing an initial interaction
`between the client and the server;
`FIG. 12 is a flow diagram showing the streaming logic of
`the server; and
`FIG. 13 is a flow diagram showing the retransmit logic of
`the server.
`
`DETAILED DESCRIPTION
`
`the invention involves a new file format for
`In short,
`organizing related multimedia information and a system and
`device for, and method of, using files organized according to
`the new format. The invention eases the management and
`control of multimedia presentations, having various media
`streams, each of a specific type, each specific type further
`classified by encoding type, subtype, and encoding rate.
`Thus, with the invention, an application may support several
`related audio streams, such as English and French, from
`wl1icl1 a user may select. Analogously, with the invention,
`the application may choose a particular instance of a media
`stream based on the network’s characteristics; for example,
`the application may choose an audio stream that is encoded
`
`

`
`5,956,729
`
`5
`for a transmission rate that matches tl1e network’s cl1arac-
`teristics. The invention reduces a server’s memory and
`processing requirements, thus allowing a server to simulta-
`neously service more requests and support more channels.
`And, the invention dynamically adapts the media’s stream-
`ing rate to use the network’s resources more efficiently while
`minimizing the effects of the adaptation on the quality of the
`presentation.
`I. The File Format
`The new file format, among other things, allows various
`types and subtypes of multimedia information to be
`organized, maintained, and used as a single file. The file
`format simplifies the server by not requiring the server to
`know, for example, which video files are related to which
`audio files for a given application, or to know how to locate
`and use related files, each with their own internal
`organization, corresponding method of access and process-
`mg.
`The file format allows multiple instances of a single
`media type to be stored in the file. Multiple instances of a
`single media type may be desirable for supporting alternate
`encodings of the same media type, for example, an audio
`segment
`in multiple languages. This flexibility allows a
`single file to contain, in elfect, multiple versions of the same
`presentation. Each instance corresponds to a “presentation’s
`worth” of information for that media type. For example,
`with the audio media type, an instance may involve the
`entire soundtrack in French or the entire soundtrack encoded
`at a particular rate.
`The file format allows media instances to be added and
`deleted to a file. This feature allows the file to be updated as
`new media types and new media segments are developed,
`without requiring modification of the servers or client’s
`logic to support the newly-added instances. This fiexibility
`makes it easier to modify, create, and maintain large, com-
`plicated multimedia presentations.
`The file format allows the server to implement more
`flexible and powerful presentations. For example, the server
`co11ld support multiple languages as various subtypes of an
`audio stream. In addition, the server could support multiple,
`expected transfer rates. For example, a video media type
`may be implemented as a subtyped instance having pre-
`packetized video data encoded for a target transfer rate of
`28.8 kb/s or encoded for a target transfer rate of 14.4 kb/s.
`Moreover, when properly used by a server or other
`application, files organized according to the new format will
`reduce the amount of memory and processor resources
`required to stream the file’s contents to a client or the like.
`These advantages are further discussed below.
`The new file format 100 is shown at a high level of
`abstraction in FIG. 1. The file format 100 includes a file
`
`header 110 and a file body 120. In short, the file header 110
`dcscribcs thc filc itsclf and thc contcnts of thc filc body 120
`and includes information used to locate data in the file body.
`The file body 120 includes more information used to locate
`data in the file body as well as including the actual data used
`during a presentation.
`the file header 110 includes a file
`More specifically,
`header preamble 210 and a number of media instance
`descriptors 220, shown in FIG. 2A. The file header preamble
`210, shown in more detail in FIG. 2B, includes a field 211
`containing a file signature, a field 212 containing the size of
`the header, a field 213 containing the major version number,
`a field 214 containing the minor version number, and a field
`215 containing the number of media instances in the file.
`(The major and minor version numbers are used for revision
`control) The file header preamble 210 also includes reserved
`fields 216 and 217 to allow for future expansion of the
`preamble.
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`As shown in more detail in FIG. 2C, each media instance
`descriptor 220 includes a variety of fields 221-225 which
`are used to describe and identify a media instance. Some of
`the fields are used to describe various characteristics or
`attributes about the instance’s data that will be presented,
`whereas other fields are used to locate and select the data.
`
`Field 221 indicating thc offsct, e.g., numbcr of bytcs, from
`the beginning of the body to the corresponding media block.
`The media instance descriptor 220 also includes a field 222
`indicating the media type of the corresponding media block,
`for example, video, audio, MIDI, and other existing and
`future media types. Field 223 indicates the encoding type of
`the corresponding media block, for example, H.263, H.261,
`MPEG, G.723, MIDI, and other standard or proprietary
`encoding types. Field 224 indicates a corresponding
`subtype, for example, English audio, French audio, QCIF
`video, CIF video, etc. Field 225 indicates an encoding rate
`of the corresponding media block; for example, for video
`information, the encoding rate indicates the target data rate
`for which the video information was encoded by a video
`encoder, such as the video encoder described in the related
`U.S. patent application “Improved Video Encoding System
`and Method”, identified and incorporated above, whereas for
`audio information, the encoding rate might indicate one of
`a number of audio sampling rates.
`As will be explained below, the number contained in field
`215 is not necessarily the same as the number of media
`streams that will eventually be involved in a presentation.
`The filc contains a numbcr of potentially rclatcd mcdia
`streams, or instances, organized according to media type,
`encoding type, subtype, and rate. A presentation, on the
`other hand, will likely involve only a subset of the available
`media streams, typically one instance of each of the plurality
`of media types. For example, a given presentation will likely
`involve only one of the multiple audio (language) subtyped
`instances that may be provided by a file organized according
`to the format. The same can be said for data encoded at
`different rates, and of course, a user may not be interested in
`a full compliment of media streams, e.g., the user may not
`be interested in receiving audio, even if it is supported by a
`file. Depending upon the services supported by the server
`(more below),
`the actual media types and particular
`instances of those media types involved in a presentation
`may be controlled by an end user, e.g., which language, and
`may also be controlled by the system, e.g., which encoded
`rate of audio.
`Once the media types and particular instances of those
`media types have been determined, for example, by being
`selected by the user or the server, the server will construct
`data structures using information from the file header 110,
`dcscribcd above, so that thc scrvcr can indcx into and itcratc
`over the data packets contained in the file body 120.
`(Indexing and iteration logic are known) The server can also
`use the header’s information to perform revision control and
`other known maintenance operations, discussed below when
`describing an exemplary server.
`The data contained in the file body 120 is organized as
`contiguous media blocks 310, one media block for each
`instance of a media type, as shown in FIG. 3. Each media
`block 310 includes a media block directory 320 and a media
`block body 330. The media block directory 320 includes
`information that may be used to locate information in the
`media block body 330, and the media block body 330
`includes tl1e actual data that will eventually be presented.
`This data is stored in the media block body 330 in pre-
`packetized form. “Pre-packetized” means that
`the data
`stored in media block body 330 is organized as discrete
`
`

`
`5,956,729
`
`7
`packets of information that can be transported without
`requiring any processing by the server to build the packets.
`An exemplary embodiment, discussed below, pre-packetizes
`the data so that
`it can be applied directly to the User
`Datagram Protocol (UDP) layer, which is part of the TCP/IP
`protocol suite. (UDP and TCP/IP are known).
`The prc-packctization process is media instancc spccific.
`For example, the G723 audio encoding standard encodes an
`audio stream into a stream of blocks, in which each block
`represents 30 milliseconds of audio information. One
`method of pre-packetizing the audio would be to form an
`audio packet for each G.723 block. A potentially more
`efficient method, however, merges many g.723 blocks into a
`packet, for example, 32 G.723 blocks to form an audio
`packet representing 960 milliseconds’ worth of audio infor-
`mation.
`
`Pre-packetizing video information, on the other hand,
`may benefit from dividing, rather than merging, presentation
`units. For example, under the H.263 video encoding
`standard, video information is encoded as a sequence of
`video frames (i.e., a frame being a presentation unit).
`Although a video packet may be formed to correspond to a
`single video frame, or presentation unit, advantages may be
`attained by dividing the presentation unit into several pack-
`ets. In this fashion, a large video frame may be divided into
`several packets so that each video packet is limited to a
`predetermined, maximum packet size.
`By pre-packetizing the data, an appropriately designed
`scrvcr’s proccssor load may bc reduced by allcviating the
`processor from having to perform certain tasks such as
`constructing packets on-the-fly from the media information.
`With the invention, the server can simply read a packet from
`the file and pass it to a UDP layer of the protocol stack via
`a standard interface.
`In addition, by pre—packetizing the data, an appropriately
`designed server’s memory requirements are by alleviating
`the server from having to keep recently transmitted packets
`available in a “packet window” in memory. Packet windows
`are conventionally used to hold recently-transmitted net-
`work packets in case they need to be retransmitted because
`they were lost in the network. The protocol being used
`dictates the required size of a packet window, but it is not
`uncommon in modern systems to have windows that require
`on the order of 100 kb of memory
`Given that each
`network channel requires a corresponding packet window
`and that
`it
`is not uncommon for current high-demand
`Internet servers to support upwards of 5,000 simultaneous
`channels (with foreseeable demand growing to over 20,000
`simultaneous channels in the near future), 512 Megabytes of
`expensive high-speed memory are needed just to support
`packct windowing. This rcquircmcnt prccludcs many mod-
`ern personal computers, and other small systems,
`from
`operating as a server. In contrast, the invention obviates the
`need for the packet windows and thus allows smaller,
`lower-cost systems to potentially operate as servers.
`The organization of a generic media block 310 is shown
`in FIGS. 4A—E and defines the basic template for a media
`block. In short, the generic media block format describes
`certain features common to all media types and instances. As
`will be described below, specific media types, such as video,
`audio, and MIDI may need to “supplement” the generic
`template.
`A generic media block directory 400, shown in FIG. 4A,
`includes a directory preamble 410 and a number of packet
`descriptors 420. The directory preamble 410, shown in more
`detail in FIG. 4B, includes a packet count field 411, indi-
`cating the number of packets contained in the media block
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`body and a presentation length field 413 indicating the total
`time duration of the presentation. A number of reserved
`fields 412 and 414 provide additional storage for media
`instance specific information (more below).
`A packet descriptor 420, shown in FIG. 4C, describes a
`single packet in the media block body. There is a one-to-one
`correspondence between packet dcscriptors 420 in thc mcdia
`block directory and packets in the media block body. Each
`packet descriptor 420 includes a packet offset field 421
`indicating the offset, e.g. bytes, from the start of the media
`block to the corresponding packet, a time stamp field 422
`indicating the start time for the packet relative to the start
`time of the presentation, an importance field 423 indicating
`the relative importance of the corresponding packet (more
`below), and a packet length field 425 indicating the length,
`e.g. bytes, of the corresponding packet.
`A generic media block body 430, shown in FIG. 4D,
`includes a number of packets 440. This number represents
`the number of packets in the stream, or instance.
`Apacket 440, shown in more detail in FIG. 4E, includes
`a channel number field 441, which is reserved for a currently
`undefined future use, and an importance field 442 indicating
`the relative importance of the packet to the perceived quality
`of the presentation (more below). Each packet 440 also
`includes fields 443-444 that may be used for segmentation
`and reassembly (SAR) of presentation units. As outlined
`above, depending on the media instance and other
`circumstances, advantages may be attained by dividing
`vidco framcs into multiple packcts. To accomplish this, ficld
`443 may be used to indicate the sequence number of the
`given packet relative to the multiple packets composing the
`presentation unit, e.g., video frame. A total segments field
`444 indicates the total number of segments.
`The packet 440 further includes a packet length field 445
`indicating the number of bytes of data contained in the
`packet, a sequence number field 447 which is reserved for a
`currently undefined future use, a time stamp field 448
`indicating the start time for the packet relative to the start of
`the presentation, and a data field 460 containing the media
`data that will be eventually presented. This media data is
`encoded in a predetermined format, according to the media
`type, encoding type, subtype, and rate (see FIG. 2C); the
`encoding format may be standardized,
`in the process of
`being standardized, or proprietary. A reserved field 449
`provides additional storage for media type specific informa-
`tion.
`includes pre-assigned
`like its descriptor,
`Each packet,
`importance information,
`indicative of the relative impor-
`tance of the packet with respect to the quality of the eventual
`presentation. As will be explained belo