(12) United States Patent
`US 6,829,648 B1
`(10) Patent N0.:
`Jones et al.
`
`(45) Date of Patent: *Dec. 7, 2004
`
`US006829648B1
`
`(54)
`
`(75)
`
`METHOD AND APPARATUS FOR
`PREPARING MEDIA DATA FOR
`TRANSMISSION
`
`Inventors: Anne Jones, Redwood City, CA (US);
`Jay Geagan, San Jose, CA (US); Kevin
`L. Gong, Sunnyvale, CA (US); Alagu
`Periyannan, San Francisco, CA (US);
`David W. Singer, San Francisco, CA
`(US)
`
`(73)
`
`Assignee: Apple Computer, Inc., Cupertino, CA
`(US)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`Appl. No.: 09/471,652
`
`Filed:
`
`Dec. 23, 1999
`
`Related US. Application Data
`
`5,497,373 A
`5,544,198 A
`5,623,490 A
`5,655,117 A
`5,659,539 A
`5,694,334 A
`
`3/1996 Hulen et al.
`8/1996 Saalfrank
`4/1997 Richter et al.
`8/1997 Goldberg et al.
`8/1997 Porter et al.
`12/1997 Donahue et al.
`
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`
`0 497 449
`0 702 309
`
`8/1992
`3/1996
`
`OTHER PUBLICATIONS
`
`P. England, et al. “RAVE: Real—time services for the Web,”
`Computer Networks and ISDN Systems 28 (1996), pp.
`1547—1558.
`
`A. Walsh, “Programming QuickTime, Multimedia to the
`Macs,” Dr. Dobbs Journal, 17:7 (Jul. 1992), XP 000600303,
`pp. 76, 78—80, 102, and 104—105.
`PCT International Search Report for PCT Int’l Appln. No.
`PCT/US99/00953 mailed Jul. 26, 1999.
`PCT International Search Report for PCT Int’l Appln. No.
`PCT/US99/00954 mailed Jul. 26, 1999.
`PCT International Search Report for PCT Int’l Appln. No.
`PCT/US99/00955 mailed Jul. 26, 1999.
`
`(21)
`
`(22)
`
`(62)
`
`(60)
`
`(51)
`(52)
`
`(58)
`
`(56)
`
`Division of application No. 09/140,173, filed on Aug. 25,
`1998, now Pat. No. 6,134,243.
`Provisional application No. 60/071,566, filed on Jan. 15,
`1998.
`
`Primary Examiner—Krisna Lim
`(74) Attorney, Agent, or Firm—Blakely, Sokoloff, Taylor &
`Zafman LLP
`
`Int. Cl.7 ................................................ G06F 15/16
`US. Cl.
`....................... 709/230; 709/231; 370/327;
`345/259
`Field of Search ................................. 370/465, 235,
`370/265, 270; 709/230, 231; 345/259, 327
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,873,777 A
`3,932,698 A
`4,688,214 A
`5,319,707 A
`5,448,568 A
`
`............ 179/15 A
`3/1975 Uehara et al.
`1/1976 Yanagimachi et al.
`....... 178/5.6
`8/1987 DeWitt et al.
`.......... 380/14
`6/1994 Wasilewski et al.
`9/1995 Delpuch et al.
`........... 372/94.2
`
`(57)
`
`ABSTRACT
`
`Methods and apparatuses for processing media data for
`transmission in a data communication medium. A set of data
`
`indicates how to transmit a time related sequence of media
`data according to a transmission protocol. The set of data,
`includes a time related sequence of data Which is associated
`With the time related sequence of media data. The set of data
`may be utilized by a digital processing system to transmit the
`time related sequence of media data (e.g., by packets gen-
`erated according to the transmission protocol and the set of
`data).
`
`81 Claims, 14 Drawing Sheets
`
`DETERMINE MEDIA FILE FORMAT
`
`{300
`
`301
`
`
`
`
`
`DETERMINE DESIRED TRANSMISSION PROTOCOLIS)
`
`
`
`CREATE AND STORE 'HINTS' FDR PACKE‘nZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA FILEIS] (PACKE‘I'IZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`305
`PROTOCOL) (HINTS AFIE NORMALLY STORED AS A TRACK OF TIME
`
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`307
` NSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`
`TIREDIA FILE(S) AND HINTS TO A RECEIVING SYSTBJI; DATA IS
`THE HIN
`TRANSMITTED BY FACKEITENG THEgEIA DATA ACCORDING TO
`
`309
`
`PRESENT, AT THE RECEIVING SYSTEM. THE MEDIAOBJECT
`REPRESENTED BY THE MEDIA DATA
`311
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
` Apple Exhibit 4426
`RECEIVING SYSTEM
`Apple v. SightSound Technologies
`CBM2013-00023
`
`Page 00001
`
`Apple Exhibit 4426
`Apple v. SightSound Technologies
`CBM2013-00023
`Page 00001
`
`

`

`US 6,829,648 B1
`
`Page 2
`
`US. PATENT DOCUMENTS
`.
`6/1998 PortueSI
`7/1998 Monteiro et a1.
`7/1998 Meyer ................... 364/400.01
`8/1998 Hamilton et a1.
`9/1998 Ludwig et a1.
`345/328
`10/1998 Throckmorton et a1.
`11/1998 DaVis et a1.
`................ 370/389
`1/1999 Porter 91 al-
`6/1999 Kouloheris 6t a1~
`7/1999 Goetz 6t a1~
`9/1999 Goetz 6t a1~
`10/1999 Arazi 6t a1~
`11/1999 Hamilton et a1.
`11/1999 Portuesi
`
`597749666 A
`5,778,187 A
`5,784,277 A
`5,799,150 A
`5,802,294 A
`5,818,441 A
`5,838,678 A
`5,864,682 A
`5,915,094 A
`59289330 A
`5,956,729 A
`5,966,120 A
`5,987,501 A
`5,987,509 A
`
`4/2000 Jones
`6,055,246 A
`5/2000 Migdal et a1.
`6,064,771 A
`8/2000 Weaver et a1.
`6,112,226 A
`9/2000 Weaver et a1.
`6,119,154 A
`10/2000 Jones et a1.
`6,134,243 A
`10/2000 Weaver et a1.
`6,138,147 A
`12/2000 Oda et a1.
`6,157,674 A
`1/2001 Schuster et a1.
`6,157,871 A1
`12/2001 Barton
`6,327,418 B1
`................. 709/230
`9/2002 Jones et a1.
`6,453,355 B1 *
`................. 370/465
`1/2003 Jones et a1.
`6,512,778 B1 *
`6/2003 Weaver et a1.
`6,578,070 B1
`7/2003 Jones et a1.
`................. 709/230
`2003/0131117 A1 *
`2003/0204555 A1 * 10/2003 Jones et a1.
`................. 709/200
`
`* cited by examiner
`
`Page 00002
`
`Page 00002
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 1 0f 14
`
`US 6,829,648 B1
`
`
`
`Page 00003
`
`Page 00003
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 2 0f 14
`
`US 6,829,648 B1
`
`ovie header
`
`video track
`
`Page 00004
`
`Page 00004
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 3 0f 14
`
`US 6,829,648 B1
`
`/ 300
`
`DETERMINE MEDIA FILE FORMAT
`
`DETERMINE DESIRED TRANSMISSION PROTOCOL(S)
`
`
`
`CREATE AND STORE 'HINTS' FOR PACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA FILE(S) (PACKETIZING IS
`
`
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`.
`‘ MEDIA DATA)
`
`
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA FILE(S) AND HINTS TO A RECEIVING SYSTEM: DATA IS
`TRANSMITTED BY PACKETIZING THE MEDIA DATA ACCORDING TO
`
`THE HINTS
`
`
`
`
`
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE' MEDIA DATA
`
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`
`RECEIVING SYSTEM
`
`FIG. 3
`
`301
`
`303
`
`305
`
`307
`
`309
`
`311
`
`Page 00005
`
`Page 00005
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 4 0f 14
`
`US 6,829,648 B1
`
`movie heads
`
`video track
`
`—-_—
`
`".
`
`I
`
`l
`
`0.
`
`5'
`
`Page 00006
`
`Page 00006
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 5 0f 14
`
`US 6,829,648 B1
`
`
`
`SimpleMovie—Video only
`
`fi-
`
`_-—-—I
`:znzl
`
`W 'l
`
`FIG. 5
`
`Page 00007
`
`Page 00007
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 6 0f 14
`
`US 6,829,648 B1
`
`602
`
`606
`
`622
`
`CLIENT
`
`COMPUTER m
`
`SYSTEM
`
`604
`CLIENT
`COMPUTER m
`
`608
`
`SYSTEM
`
`610
`
`6533A“
`
`612
`
`SYSTEM
`
`VVEB .
`SERVER
`
`INTERNET
`
`\
`
`600
`
`614
`
`'
`
`616
`
`NETWORK
`INTERFACE
`
`NETWORK
`INTERFACE
`
`CLIENT
`COMPUTER
`SYSTEM
`
`CLIENT
`COMPUTER
`SYSTEM
`
`618
`
`620
`
`FIG. 6
`
`Page 00008
`
`Page 00008
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 7 0f 14
`
`US 6,829,648 B1
`
`
`
`
`
`
`664
`
`652
`
`PROCESSOR
`
`'
`
`658
`
`DISPLAY
`CONTROLLER ,
`
`MASS
`MEMORY
`
`662
`
`I/O
`CONTROLLER
`
`'
`
`660
`
`v0
`
`DIGITAL PROCESSING
`SYSTEM
`
`650
`—
`
`
`
`
`MODEM
`OR
`NETWORK
`INTERFACE
`
`668
`
`FIG. 7
`
`Page 00009
`
`Page 00009
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 8 0f 14
`
`US 6,829,648 B1
`
`MEDIA
`
`PROCESSING
`UNIT
`
`684
`
`HINT GENERATION
`AND PROCESSING UNIT
`
`MEDIA PROCESSING
`UNIT
`
`SERVER gag
`
`CLIENT DATA
`PROCESSING SYSTEM
`682
`——
`
`DATA
`COMMUNICAT'ON
`LINK ass
`
`DATA COMMUNICATION
`UNIT
`
`SYSTEM 680 '1
`
`FIG. 8
`
`Page 00010
`
`Page 00010
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 9 0f 14
`
`US 6,829,648 B1
`
`
`
`MEDIA
`
`
`PROCESSING
`
`
`UNIT
`
`
`
`
`‘ 684
`
`
`CLIENT DATA
`DATA
`PROCESSING SYSTEM
`COMMUNICATION
`
`.653
`LINK 686
`
`
`
`
`702
`
`HINT PROCESSING
`UNIT '
`
`'
`
`‘ 704
`
`MEDIA PROCESSING
`UNIT
`
`706
`
`DATA COMMUNICATION
`UNIT
`
`SERVER m
`
`
`
`
`
`
`
`
`
`
`
`
`
`GENERATOR _7_1_Q
`
`SYSTEM 696
`
`/
`
`DATA
`
`COMMUNICATION
`UN" 708
`
`712
`
`HINT GENERATION
`UNIT
`
`FIG. 9
`
`Page 00011
`
`Page 00011
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 10 0f 14
`
`US 6,829,648 B1
`
`DETERMINE MEDIA FORMAT (IF MORE THAN
`ONE FORMAT IS/WILL BE USED)
`
`720
`
`DETERMINE DESIRED DATA COMMUNICATION
`PROTOCOL(S) (IF MORE THAN ONE PROTOCOL
`IS/WILI. BE USED)
`
`
`
`BASED ON THE MEDIA FORMAT AND DESIRED
`DATA COMMUNICATION PROTOCOLIS), CREATE
`AND STORE HINTS FOR PACKETIZING A
`TIME RELATED SEQUENCE OF MEDIA DATA
`
`722
`
`724
`
`OPTIONALLY TRANSMIT HINTS TO ANOTHER
`'
`DIGITAL PROCESSING SYSTEM
`
`726
`
`FIG. 10
`
`Page 00012
`
`Page 00012
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 11 0f 14
`
`US 6,829,648 B1
`
`RECEIVE, AT A RECEIVING SYSTEM,
`MEDIA DATA TRANSMITTED TO THE
`RECEIVING SYSTEM IN ACCORDANCE
`
`WITH "HINTS‘l
`
`,
`
`PRESENT, AT THE RECEIVING SYSTEM,
`A MEDIA OBJECT REPRESENTED BY
`
`THE MEDIA DATA
`
`OPTIONALLY STORE THE MEDIA DATA
`AS A MEDIA FILE
`
`730
`
`732
`
`734
`
`OPTIONALLY REASSEMBLE THE STORED
`-
`MEDIA FILE .
`
`736
`
`FIG. 11
`
`Page 00013
`
`Page 00013
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 12 0f 14
`
`US 6,829,648 B1
`
`742
`
`
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`
`
`
`
`
`
`INFORMATION RELATING
`TO ONE OR MORE DATA
`COMMUNICATION PROTOCOLS
`AND/OR MEDIA FORMATS
`
`
`
`
`
`MACHINE-READABLE
`STORAGE MEDIUM
`740
`
`FIG. ‘I 2
`
`'
`
`-
`
`GENERATOR
`OS
`
`HINT CREATION
`ROUTINE(S)
`
`CREATED HINTS
`
`SERVER OS
`
`MEDIA DATA
`
`744
`
`746
`
`762
`
`764
`
`766
`
`.
`
`
`
`
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`
`
`'
`MEDIA PACKETIZATION
`ROUTINE(S), BASED
`ON HINTS
`7
`
`
`
`“ms '
`
`'
`
`
`
`
`MACHINE-READABLE
`STORAGE MEDIUM
`fl
`
`FIG. 13
`
`Page 00014
`
`Page 00014
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 13 0f 14
`
`US 6,829,648 B1
`
`772
`
`RECEIVING SYSTEM
`'. OS
`
`
`
`778
`
`MEDIA PRESENTATION
`ROUTINE(S)
`
`(OPTIONAL)
`MEDIA DATA
`
`-
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`782
`
`774
`
`776
`
`
`
`
`
`
`
`
`
`
`
`
`
`STORAGE MEDIUM
`fi'gfmfi},
`FIE-ASSEMBLY ROUTINE
`.
`.Z§.9.
`
`
`MACHINE-READABLE
`
`FIG. 14
`
`Page 00015
`
`Page 00015
`
`

`

`US. Patent
`
`Dec. 7, 2004
`
`Sheet 14 0f 14
`
`US 6,829,648 B1
`
`HINT PACKET
`
`MEDIA DATA PACKET
`
`£0.
`
`~ DATA STORAGE AND/OR
`COMMUNICATION MEDIUM
`
`FIG. 15
`
`Page 00016
`
`Page 00016
`
`

`

`1
`METHOD AND APPARATUS FOR
`PREPARING MEDIA DATA FOR
`TRANSMISSION
`
`This application is a divisional application of Ser. No.
`09/140,173, now US. Pat. No. 6,134,243, filed Aug. 25,
`1998, which is a continuation-in-part of Serial No. 60/071,
`566, filed Jan. 15, 1998.
`
`FIELD OF THE INVENTION
`
`The present invention relates to methods and apparatuses
`for preparing time related sequences of media data for
`transmission, and more particularly to packetized transmis-
`sion of such media data.
`
`INTRODUCTION AND BACKGROUND
`
`There are various different file structures used today to
`store time-based media: audio formats such as AIFF, video
`formats such as AVI, and streaming formats such as Real-
`Media. One reason that such file structures are different is
`
`their different focus and applicability. Some of these formats
`are sufficiently relatively widely accepted, broad in their
`application, and somewhat simple to implement, and thus,
`may be used not only for content delivery but also as
`interchange formats. Foremost among these general formats
`is the QuickTime file format. It is used today in the majority
`of web sites serving time-based data;
`in the majority of
`authoring environments, including professional ones; and on
`the majority of multimedia CDROM titles.
`The QuickTime media layer supports the efficient display
`and management of general multimedia data, with an
`emphasis on time-based material (video, audio, etc.). The
`media layer uses the QuickTime file format as the storage
`and interchange format for media information. The archi-
`tectural capabilities of the layer are generally broader than
`the existing implementations, and the file format is capable
`of representing more information than is currently
`demanded by the existing QuickTime implementations.
`In contrast to formats such as AVI, which were generally
`designed to support local random access of synchronized
`media, QuickTime allows systems to manage the data,
`relationships and timing of a general multimedia presenta-
`tion. In particular, the QuickTime file format has structures
`to represent the temporal behavior of general time-based
`streams, a concept which covers the time-based emission of
`network packets, as well as the time-based local presentation
`of multimedia data.
`
`The existing QuickTime file format is publicly described
`by Apple Computer
`in the May 1996 File format
`specification, which may be found at the QuickTime site,
`<http://.www.apple.com/quicktime>.
`One aspect of the QuickTime file format is the concept
`that the physical structure of media data (the layout in disk
`records) is independent of, and described by, a logical
`structure for the file. The file is fully described by a set of
`“movie” meta-data. This meta-data provides declarative,
`structural and temporal information about the actual media
`data.
`
`The media data may be in the same file as the description
`data, (the “movie” meta-data), or in other file(s). A movie
`structured into one file is commonly called “flat”,and is
`self-contained. Non-fiat movies can be structured to refer-
`ence some, or all, of the media data in other files.
`As such, the format is generally suited for optimization in
`different applications. For example, when editing
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,829,648 B1
`
`2
`
`(compositing), data need not be rewritten as edits are applied
`and media is reordered; the meta-data file may be extended
`and temporal mapping information adjusted. When edits are
`complete, the relevant media data and meta-data may be
`rewritten into a single, interleaved, and optimized file for
`local or network access. Both the structured and the opti-
`mized files are valid QuickTime files, and both may be
`inspected, played, and reworked.
`The use of structured (“non-flat”) files enables the same
`basic media data to be used and re-used in any number of
`presentations. This same advantage applies when serving, as
`will be seen below.
`
`In both editing and serving, this also permits a number of
`other files to be treated as part of a movie without copying
`the media data. Thus editing and serving may be done
`directly from files such as Sun Microsystem’s “au” audio
`format or the AVI video format, greatly extending the utility
`of these formats.
`
`The QuickTime file is divided into a set of objects, called
`atoms. Each object starts with an atom header, which
`declares its size and type:
`
`class Atom {
`
`int(32)
`char
`byte
`
`size;
`type[4];
`contents[ ];
`
`The size is in bytes, including the size and type header
`fields. The type field is four characters (usually printable), to
`permit easy documentation and identification. The data in an
`object after the type field may be fields, a sequence of
`contained objects, or both.
`A file therefore is simply a sequence of objects:
`
`class File {
`Atom[ ];
`
`}
`
`The two important top-level objects are the media-data
`(mdat) and the meta-data (moov).
`The media-data object(s) contain the actual media (for
`example, sequences of sound samples). Their format is not
`constrained by the file format; they are not usually objects.
`Their format
`is described in the meta-data, not by any
`declarations physically contiguous with them. So,
`for
`example,
`in a movie consisting solely of motion-JPEG,
`JPEG frames are stored contiguously in the media data with
`no intervening extra headers. The media data within the
`media data objects is logically divided into chunks;
`however, there are no explicit chunk markers within the
`media data.
`
`When the QuickTime file references media data in other
`files, it is not required that these ‘secondary’ files be for-
`matted according to the QuickTime specification, since such
`media data files may be formatted as if they were the
`contents of a media object. Since the QuickTime format does
`not necessarily require any headers or other information
`physically contiguous with the media data, it is possible for
`the media data to be files which contain ‘foreign’ headers
`(e.g. UNIX “.au” files, or AVI files) and for the QuickTime
`meta-data to contain the appropriate declarative information
`and reference the media data in the ‘foreign’ file. In this way
`
`Page 00017
`
`Page 00017
`
`

`

`US 6,829,648 B1
`
`3
`the QuickTime file format can be used to update, without
`copying, existing bodies of material in disparate formats.
`The QuickTime file format is both an established format and
`is able to work with, include, and thereby bring forward,
`other established formats.
`
`Free space (e.g. deleted by an editing operation) can also
`be described by an object. Software reading a file that
`includes free space objects should ignore such free space
`objects, as well as objects at any level which it does not
`understand. This permits extension of the file at virtually any
`level by introducing new objects.
`The primary meta-data is the movie object. A QuickTime
`file has exactly one movie object which is typically at the
`beginning or end of the file, to permit its easy location:
`
`class Movie {
`int(32)
`char
`MovieHeader
`contents
`
`size;
`type[4] = ‘moov’;
`mh;
`Atom[ ];
`
`10
`
`15
`
`20
`
`4
`At the lowest level, a sample table is used which relates
`the temporal aspect of the track to the data stored in the file:
`
`class sampletable {
`int(32)
`char
`sampledescription
`timetosample
`syncsampletable
`sampletochunk
`samplesize
`chunkoffset
`shadowsync
`
`size;
`type[4] = ‘ stbl’;
`sd;
`tts;
`syncs;
`stoc;
`ssize;
`coffset;
`ssync;
`
`The sample description contains information about the
`media (e.g.
`the compression formats used in video). The
`time-to-sample table relates time in the track, to the sample
`(by index) which should be displayed at that time. The sync
`sample table declares which of these are sync (key) samples,
`not dependent on other samples.
`
`The movie header provides basic information about the
`overall presentation (its creation date, overall timescale, and
`so on). In the sequence of contained objects there is typically
`at least one track, which describes temporally presented
`data.
`
`The sample-to-chunk object declares how to find the
`media data for a given sample, and its description given its
`index:
`
`25
`
`class sampletochunk {
`int(32)
`size;
`char
`type[4] = ‘stsc’;
`int(8)
`version;
`bits(24)
`flags;
`int(32)
`entrycount;
`for (int i=0; i<entrycount; i++) {
`int(32)
`firstchunk;
`int(32)
`samplesperchunk;
`int(32)
`sampledescriptionindex;
`
`}
`
`}
`
`The sample size table indicates the size of each sample.
`The chunkoffset table indicates the offset into the containing
`file of the start of each chunk.
`
`Walking the above-described structure to find the appro-
`priate data to display for
`a given time is fairly
`straightforward, generally involving indexing and adding.
`Using the sync table, it is also possible to back-up to the
`preceding sync sample, and roll forward ‘silently’ accumu-
`lating deltas to a desired starting point.
`
`FIG. 1 shows the structure of a simple movie with one
`track. Asimilar diagram may be found in the QuickTime file
`format documentation, along with a detailed description of
`the fields of the various objects. QuickTime atoms (objects)
`are shown here with their type in a grey box, and a
`descriptive name above. This movie contains a single video
`track. The frames of video are in the same file, in a single
`chunk of data. It should be noted that the ‘chunk’ is a logical
`construct only;
`it
`is not an object. Inside the chunk are
`frames of video, typically stored in their native form. There
`are no required headers or fields in the video frames them-
`selves.
`
`FIG. 2 is a diagram of a self-contained file with both an
`audio and a video track. Fewer of the atoms are shown here,
`for brevity; the pointers from the tracks into the media data
`are, of course, the usual sample table declarations, which
`include timing information.
`
`Page 00018
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`class Track {
`int(32)
`char
`TrackHeader
`contents
`
`size;
`type[4] = ‘trak’;
`th;
`Atom[ ];
`
`The track header provides relatively basic information
`about
`the track (its ID,
`timescale, and so on). Objects
`contained in the track might be references to other tracks
`(e. g. for complex compositing), or edit lists. In this sequence
`of contained objects there may be a media object, which
`describes the media which is presented when the track is
`played.
`The media object contains declarations relating to the
`presentation required by the track (e.g. that it is sampled
`audio, or MIDI, or orientation information for a 3Dscene).
`The type of track is declared by its handler:
`
`class handler {
`int(32)
`char
`int(8)
`bit(24)
`char
`char
`
`audio
`
`size;
`type[4] = ‘hdlr’;
`version;
`flags;
`handlertype[4]; -- mhlr for media handlers
`handlersubtype[4] -- vide for video, soun for
`
`char
`bit(32)
`bit(32)
`string
`
`manufacturer[4];
`handlerfiags;
`handlerfiagsmask;
`componentname;
`
`ithin the media information there is likewise a handler
`
`} W
`
`declaration for the data handler (which fetches media data),
`and a data information declaration, which defines which files
`contain the media data for the associated track. By using this
`declaration, movies may be built which span several files.
`
`Page 00018
`
`

`

`US 6,829,648 B1
`
`5
`The QuickTime file format has a number of advantages,
`including:
`1) Scalability for size and bit-rates. The meta data is
`flexible, yet compact. This makes it suitable for small
`downloaded movies (e.g. on the Internet) as well as
`providing the basis for a number of high-end editing
`systems.
`2) Physical structure is independent of the logical and
`temporal structure. This makes it possible to optimize
`the physical structure differently depending on the use
`the file will have. In particular, it means that a single file
`format is suitable for authoring and editing; download-
`ing or placing on CDROMs; and for streaming.
`3) The file format has proven capable of handling a very
`broad variety of codec types and track types, including
`many not known at the time the format was designed.
`This proven ability to evolve in an upwards-compatible
`fashion is fundamental
`to the success of a storage
`format.
`
`Scalable, or layered codecs can be handled in a number of
`ways in the QuickTime file format. For a streaming protocol
`which supports scalability, the samples may be tagged with
`the layer or bandwidth threshold to be met for transmitting
`the samples.
`Tracks which form a set of alternatives (e.g. different
`natural language sound tracks) can be tagged so that only
`one is selected for playback. The same structure can be used
`to select alternatives for streaming (e.g.
`for
`language
`selection). This capability is described in further detail in the
`QuickTime file format.
`When QuickTime displays a movie or track, the appro-
`priate media handler accesses the media data for a particular
`time. The media handler must correctly interpret the data
`stream to retrieve the requested data. For example, with
`respect to video media, the media handler typically traverses
`several atoms to find the location and size of a sample for a
`given media time. The media handler may perform the
`following:
`1. Determine the time in the media time coordinate
`system.
`2. Examine the time-to-sample atom to determine the
`sample number that contains the data for the specified
`time.
`
`3. Scan the sample-to-chunk atom to discover which
`chunk contains the sample in question.
`4. Extract the offset to the chunk from the chunk offset
`atom.
`
`5. Find the offset within the chunk and the sample’s size
`by using the sample size atom.
`It is often desirable to transmit a QuickTime file or other
`types of time related sequences of media data over a data
`communication medium, which may be associated with a
`computer network (e.g.
`the Internet). In many computer
`networks, the data which is transmitted into the network
`should generally be in a packet form. Normally, time related
`sequences of media data are not in the proper packetized
`format for transmission over a network. For example, media
`data files in the QuickTime format are not in a packetized
`format. Thus, there exists a need to collect the data, some-
`times referred to as streaming data, into packets for trans-
`mission over a network.
`
`One prior approach to address the problem of transmitting
`time related sequences of media data over a network is to
`send the media file over the network using a network or
`transmission protocol, such as the Hypertext Transfer Pro-
`tocol (HTTP). Thus, the media file itself is sent from one
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`computer system over the network to another computer
`system. However, there may be no desire to retain the media
`file at the receiving computing system. That is, when the
`media file is received and viewed or listened to at
`the
`
`receiving computer system, there may be no desire by the
`user of that receiving computer system to store a copy of the
`file, for example, if the receiving computing system is a
`network computer or a computer with low storage capacity.
`Another alternative approach to solving the problem of
`how to collect data for transmission by packets over a
`network is to prepare a file which contains the network
`protocol data units in the file for a particular transmission
`protocol.
`In a sense, such a file may be considered a
`packetized file which is stored in essentially the same format
`as it will be transmitted according to the particular trans-
`mission protocol. Performing it,
`this operation generally
`involves storing the file in a packetized form for a particular
`network protocol at a particular data transmission rate and a
`particular media file format. Thus, for each different trans-
`mission protocol at a particular data transmission rate, the
`file will essentially be replicated in its packetized form. The
`fixed form of such files may restrict
`their applicability/
`compatibility and make it difficult to view such files locally.
`Thus, such an approach may greatly increase storage
`requirements in attempting to provide the file in various
`transmission protocols at various different data transmission
`rates. Moreover, each packetized file generated according to
`this alternative prior approach is generally limited to a
`particular media file format, and thus, other media file
`formats for the same media object (e.g. a digital movie) are
`typically packetized and stored on the sending computer
`system.
`Yet another approach to solving the problem of how to
`stream time related sequences of media data is to perform
`the packetization of the media data when required on the
`transmitting system according to the particular transmission
`protocol which is desired. This processing requires, in many
`cases, a relatively considerable amount of time, and thus,
`may slow the performance of the transmitting system.
`Thus, it is desirable to provide an improved method and
`apparatus for transmitting time related sequences of media
`data.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides methods and apparatuses
`for processing media data for transmission in a data com-
`munication medium. In one embodiment, a set of data
`indicates how to transmit a time related sequence of media
`data according to a transmission protocol. The set of data,
`according to one embodiment,
`includes a time related
`sequence of data which is associated with the time related
`sequence of media data According to one aspect of the
`invention,
`the set of data may be utilized by a digital
`processing system to transmit the time related sequence of
`media data (e.g., by packets generated according to the
`transmission protocol and the set of data).
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 shows an example of the structure of a simple
`movie with one track in the prior art.
`FIG. 2 is an example of a self-contained movie file of the
`prior art.
`FIG. 3 is a flowchart showing one example of a method
`according to the present invention.
`FIG. 4 shows an example of a hint track of the present
`invention.
`
`Page 00019
`
`Page 00019
`
`

`

`US 6,829,648 B1
`
`7
`FIG. 5 shows another example of a hint track of the
`present invention.
`FIG. 6 is a diagram of a network of computer systems in
`which media data may be exchanged and/or processed,
`according to one embodiment of the present invention.
`FIG. 7 is a block diagram of a digital processing system
`which may be used in accordance with one embodiment of
`the present invention.
`FIG. 8 is a block diagram of a system that utilizes hints
`to transfer media data, according to one embodiment of the
`invention.
`
`FIG. 9 is a block diagram of a system that utilizes hints
`to transfer media data, according to one embodiment of the
`invention.
`
`FIG. 10 is a flow diagram illustrating a method for
`generating hints for providing media data transmission,
`according to one embodiment of the invention.
`FIG. 11 is a flow diagram illustrating a method of pro-
`cessing media data received by a receiving system in accor-
`dance with hints, according to one embodiment of the
`invention.
`
`FIG. 12 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a generator, according to one embodiment of
`the invention.
`
`FIG. 13 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a server, according to one embodiment of the
`invention.
`
`FIG. 14 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a receiving system or other digital process-
`ing system, according to one embodiment of the invention.
`FIG. 15 is a diagram of a data storage and/or communi-
`cation medium having stored/transported thereon media and
`hint information, according to one embodiment of the inven-
`tion.
`
`DETAILED DESCRIPTION
`
`The present invention provides methods and apparatuses
`for allowing the transmission, and particularly the pack-
`etized transmission of time related sequences of media data,
`which may include, for example, video, audio, video and
`audio, etc., over a communication media, such as in a
`computer network.
`In one embodiment of the present invention, a digital
`processing system creates a set of data for indicating how to
`transmit a time related sequence of media data according to
`a transmission protocol. Typically, this set of data is stored
`on a storage device coupled to the digital processing system.
`Further, this set of data is a time related sequence of data
`associated with the time related sequence of media data
`The present invention may be implemented entirely in
`executable computer program instructions which are stored
`on a computer readable media or may be implemented in a
`combination of software and hardware, or
`in certain
`embodiments, entirely in hardware. Typically, a server com-
`puter system coupled to a network will create the set of data,
`which may be referred to as a hint track and will store this
`hint track in a storage device which is coupled to the server
`computer system. When a client computer system requests a
`presentation (e.g. a viewing or listening or viewing and
`listening) of a media data file, the server system uses the hint
`track to determine how to packetize the media data for
`transmission to the client computer system. It will be appre-
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`ciated that the present invention is generally applicable to
`time related sequences of media data, and that QuickTime is
`represented herein as one example of this general applica-
`bility. Thus, the invention should not necessarily be limited
`to QuickTime.
`FIG. 3 shows one example of a method according to the
`present invention. The method 300 shown in FIG. 3 begins
`in step 301, in which the media file format for the particular
`media data which is desired to be transmitted is determined.
`In step 303, the particular transmission protocol or protocols
`which are desired to be used is also determined. However,
`steps 301 and 303 are optional, for example, in the case
`where the same media file format is always transmitted
`using the same transmission protocol.
`In step 305, a digital processing system, such as a server
`computer system, creates and stores the hints for packetizing
`a time related sequence of media data in a media file.
`Alternatively, one computer system may create the hints and
`provide them to another system, such as a server computer
`system, which stores them for later use in a transmission
`process. The packetization allows the transmission over a
`network or communication media according to the desired
`transmission protocol which was determined in step 303. In
`one embodiment of the present
`invention,
`the hints are
`stored as a track of time related sequence of hints which
`refers to, but which in one embodiment, is separate from
`other tracks of media data. The track of hints,
`in one
`embodiment of the present invention, may be stored sepa-
`rately from the media data to which it refers. As such, the
`track of hints may be stored in a file which is distinct from
`another file containing the media data which is referred to by
`the track of hints, or the track of hints may be stored in a hint
`area in th