`Jones et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,717,952 B2
`*Apr. 6, 2004
`
`US006717952B2
`
`(54) METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`(56)
`
`References Cited
`U.s. PATENT DOCUMENTS
`
`(75)
`
`IHVWOFSI Anne J0I1eS> R€dW00d City, CA (US);
`Jay Geagan, San Jose, CA (US); Kevin
`L. Gong, Sunnyvale, CA (US); Alagu
`Periyannan, Fremont, CA (US); David
`W_ Singer’ San Francisco’
`
`3,873,777 A
`3,932,698 A
`4,688,214 A
`5,319,707 A
`A
`
`.......... .. 179/15 A
`3/1975 Uehara et al.
`1/1976 Yanagimachi et al.
`..... .. 178/5.6
`8/1987 DeWitt et al.
`........ .. 380/14
`6/1994 Wasilewski et al.
`Delpuch CI al.
`. . . . . . . . . ..
`
`(73) Assignee: Apple Computer, Inc., Cupertino, CA
`(US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`EP
`EP
`
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`
`H04B/7/212
`......... .. G06F/17/00
`
`0497449
`0702309
`
`8/1992
`3/1996
`
`OTHER PUBLICATIONS
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21) Appl. No.: 10/298,039
`
`(22)
`
`Filed:
`
`Nov. 14, 2002
`
`(65)
`
`Prior Publication Data
`US 2003/0204555 A1 Oct. 30, 2003
`
`P. England, et al. “RAVE: Real-time services for the Web,”
`Computer Networks and ISDN Systems 28 (1996), pp.
`1547-1558.
`
`A. Walsh, “Programming Quick Time, Multimedia to the
`Macs,” Dr. Dobbs Journal, 17:7 (Jul. 1992), XP 000600303,
`pp. 76, 78-80, 102, and 104-105.
`PCT International Search Report for PCT Int’l Appln. No.
`PCT/US99/00953 mailed Jul. 26, 1999.
`
`(List continued on next page.)
`
`Related U.S. Application Data
`
`Primary Examiner—Dang Ton
`Assistant Examiner—Shick Horn
`
`(63) Continuation of application No. 09/651,009, filed on Aug.
`29, 2000, now Pat. No. 6,512,778, which is a continuation
`of application No. 09/140,173, filed on Aug. 25, 1998, now
`Pat. No. 6,134,243.
`Provisional application No. 60/071,566, filed on Jan. 15,
`1998.
`
`(60)
`
`(51)
`
`Int. Cl.7 ................................................. .. H04J 3/16
`
`(52) U.s. Cl.
`
`..................... .. 370/465; 370/235, 370/265;
`370/270, 370/395.52, 370/395.6; 370/487;
`709/230
`
`(58) Field of Search ............................... .. 370/235, 259,
`370/263, 265, 394, 395.52, 465, 466, 467,
`468, 486, 487, 270, 395.6; 709/203, 230,
`231, 232
`
`(74) Attorney, Agent, or Firm—Blakely, Sokoloff, Taylor &
`Zafman LLP
`
`(57)
`
`ABSTRACT
`
`Methods and apparatuses for processing media data for
`transmission in a data communication medium. A set of data
`
`indicates how to transmit a time related sequence of media
`data according to a transmission protocol. The set of data,
`includes a time related sequence of data which is associated
`with the time related sequence of media data. The set of data
`may be utilized by a digital processing system to transmit the
`time related sequence of media data (e.g., by packets gen-
`erated according to the transmission protocol and the set of
`data).
`
`54 Claims, 14 Drawing Sheets
`
`5,0‘
`
`303
`
`307
`
`309
`
`31 l
`
`/ 300
`
`DETERMINE MEDIA FILE FORMAT
`
`DETERMINE DESIRED TRANSMISSION PROTOCOL(S)
`
`CREATE AND STORE "HINTS" FOR FACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA F|LE(SI (PACKETIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLV STORED AS A TRACK OF TIME
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA FILE(S) AND HINTS TO A RECEIVING SVSTEM; DATA IS
`TRANSMITTED EV PACKETIZING THE MEDIA DATA ACCORDING TO
`THE HINTS
`
`
`
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`OPTIONALLV REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`RECEIVING SYSTEM
`
`Apple Exhibit 4425
`
`Apple v. SightSound Technologies
`CBM2013-00023
`
`Page 00001
`
`Apple Exhibit 4425
`Apple v. SightSound Technologies
`CBM2013-00023
`Page 00001
`
`
`
`US 6,717,952 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`3/1996 Hrr1errete1~
`5,497,373 A
`8/1996 Saalfrank .................. .. 370/343
`5,544,198 A *
`4/1997 Riehter eta1~
`5,623,490 A
`............ .. 395/615
`4/1997 Zarmer et al.
`5,625,818 A *
`8/1997 Goldberg eta1~
`5,655,117 A
`12/1997 Derrahrre et 61-
`5,694,334 A
`6/1998 Portuesi
`5,774,666 A
`7/1998 Merrteire eta1~ ~~~~~~~~~ ~~ 370/351
`5,778,187 A *
`7/1998 Meyer ~~~~~~~~~~~~~~~~~ ~~ 364/40001
`5,784,277 A
`8/1998 Hamilton et 211.
`5,799,150 A
`........... .. 370/267
`9/1998 Ludwig et al.
`5,802,294 A *
`5,818,441 A * 10/1998 Throckmorton et al.
`345/328
`2 13 3911.5 et ‘SL1 """""""" 325833
`,
`,
`er ins e a.
`............... ..
`5,915,094 A *
`6/1999 Kouloheris et al.
`....... .. 370/486
`5,928,330 A
`7/1999 Goetz et a1.
`5,956,729 A
`9/1999 Goetz et al.
`5,966,120 A
`10/1999 Arazietal.
`5,987,501 A * 11/1999 Hamilton et al.
`
`......... .. 709/203
`
`11/1999 Portuesi
`5,987,509 A
`............ .. 370/263
`5,995,491 A * 11/1999 Richter et al.
`6,055,246 A *
`4/2000 Jones ....................... .. 370/503
`6,064,771 A
`5/2000 Migdai et al.
`6,134,243 A * 10/2000 Jones et al.
`............... .. 370/465
`6,157,674 A
`12/2000 Oda et al.
`6,175,871 B1
`1/2001 Schuster et al.
`6,175,872 B1
`1/2001 Neumann et al.
`6,327,418 B1
`12/2001 Barton
`6,453,355 B1 *
`9/2002 Jones et al.
`6,512,778 B1 *
`1/2003 Jones etal.
`
`............... .. 370/259
`............... .. 370/465
`
`......... .. 709/231
`
`OTHER PUBLICATIONS
`
`PCT International Search Report for PCT Int’l Appln. No.
`-
`PCT/U599/(4)954 mafled Jul‘ 26’ 1999'
`PCT International Search Report for PCT Int’l Appln. No.
`PCT/US99/00955 mailed Jul. 26, 1999.
`
`* cited by examiner
`
`Page 00002
`
`Page 00002
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 1 of 14
`
`US 6,717,952 B2
`
`
`
`
`
`D
`
`D
`
` media data
`
`D D
`
`E! El
`
`D D
`
`E! El
`
`E1 E1
`
`~El E!
`
`E!
`
`El
`
`I I I
`
`FIG. 1
`
`Prior Art
`
`Page 00003
`
`Page 00003
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 2 of 14
`
`US 6,717,952 B2
`
`movie header
`
`(video) track
`
`Page 00004
`
`Page 00004
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 3 of 14
`
`US 6,717,952 B2
`
`/ 300
`
`DETERMINE MEDIA FILE FORMAT
`
`DETERMINE DESIRED TRANSMISSION PROTOCOL(S)
`
`CREATE AND STORE "HINTS" FOR PACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA F|LE(S) (PACKETIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`
`
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA F|LE(S) AND HINTS TO A RECEIVING SYSTEM; DATA IS
`TRANSMITTED BY PACKETIZING THE MEDIA DATA ACCORDING TO
`THE HINTS
`
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`
`RECEIVING SYSTEM
`
`FIG. 3
`
`301
`
`303
`
`305
`
`307
`
`309
`
`311
`
`Page 00005
`
`Page 00005
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 4 of 14
`
`US 6,717,952 B2
`
`movie heade
`
`(video) track
`
`trak
`
`FIG. 4
`
`Page 00006
`
`Page 00006
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 5 of 14
`
`US 6,717,952 B2
`
`Simple Movie—Video only
`
`
`
`
`
`_ fl -
`
`Hint Movi Videometa-d
`
` —_
`
`
`
`__——=
`11-1-
`
`
`RTF’ hint
`track—_-
`:—:—
`
`
`—)—_-
`RTPPackethint
`RTPPackethig
`RTPPackethipt
`E: I u
`.
`samp
`sampe
`
`'”
`
`d t
`
`
`
`
`504
`
`FIG. 5
`
`Page 00007
`
`Page 00007
`
`
`
`U.S. Patent
`
`Apr. 6,2004
`
`Sheet 6 of 14
`
`US 6,717,952 B2
`
`602
`
`CLIENT
`
`COMPUTER
`SYSTEM
`
`606
`
`MODEM
`
`604
`
`
`
`CLIENT
`
`
`COMPUTER
`
`SYSTEM
`
`
`61 O
`
`. GATEWAY
`SYSTEM
`
`612
`
`614
`
`616
`
`NETWORK
`INTERFACE
`
`NETWORK
`INTERFACE
`
`CLIENT
`
`COMPUTER
`SYSTEM
`
`CLIENT
`
`COMPUTER
`SYSTEM
`
`618
`
`620
`
`FIG. 6
`
`INTERN ET
`
`WEB
`
`SERVER
`
`SYSTEM
`
`,\
`
`600
`
`Page 00008
`
`Page 00008
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 7 of 14
`
`US 6,717,952 B2
`
`652
`
`PROCESSOR
`
`MEMORY
`
`554
`
`656
`
`658
`
`
`
`
`
`
`
`DISPLAY
`CONTROLLER
`
`MASS
`MEMORY
`
`I/O
`CONTROLLER
`
`664
`
`662
`
`DISPLAY
`
`550
`
`IIO
`DEVICES
`
`666
`
`DIGITAL PROCESSING
`SYSTEM
`
`95.0.
`
` MODEM
`
`
`
`OR
`
` 668
`
`NETWORK
`
`
`INTERFACE
`
`
`FIG. 7
`
`Page 00009
`
`Page 00009
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 8 of 14
`
`US 6,717,952 B2
`
`688
`
`
`
`
`MEDIA
`PROCESSING
`UNIT
`
`
`
`
`
`
`
`CLIENT DATA
`PROCESSING SYSTEM
`COMMEWQATION
`
`_6_§;
`
`
`LINK 686
`
`
`
`
`684
`
`
`
`HINT GENERATION
`AND PROCESSING UNIT
`590
`
`MEDIA PROCESSING
`UNIT
`
`2
`
`69
`
`DATA COMMUNICATION
`UNIT
`
`
`
`
`
`SERVER ggg
`
`SYSTEM 680 ]
`
`FIG. 8
`
`Page 00010
`
`Page 00010
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 9 of 14
`
`US 6,717,952 B2
`
`
`
`
`
`
`MEDIA
`
`PROCESSING
`UNIT
`
`702
`
`HINT PROCESSING
`UNIT
`
`704
`
`MEDIA PROCESSING
`
`UNIT
`
`706
`
`DATA COMMUNICATION
`
`UNIT
`
`SERVER 100
`
`
`
`
`
`
`
`PROCESSING SYSTEM
`LINK 686
`fig
`COM
`
`MUNICATION
`
`
`
`CLIENT DATA
`
` 684
`
`
`DATA
`
`SYSTEM 696
`
`1
`
`DATA
`
`COMMUNICATION
`LINK 708
`
`
`
`712
`
`HINT GENERATION
`UNIT
`
`
`
`
`GENERATOR _7_1Q
`
`
`
`FIG. 9
`
`Page 00011
`
`Page 00011
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 10 of 14
`
`US 6,717,952 B2
`
`DETERMINE MEDIA FORMAT (IF MORE THAN
`ONE FORMAT IS/WILL BE USED)
`
`720
`
`DETERMINE DESIRED DATA COMMUNICATION
`PROTOCOL(S) (IF MORE THAN ONE PROTOCOL
`ISNVILL BE USED)
`
`
`
`BASED ON THE MEDIA FORMAT AND DESIRED
`DATA COMMUNICATION PROTOCOL(S), CREATE
`AND STORE HINTS FOR PACKETIZING A
`
`TIME RELATED SEQUENCE OF MEDIA DATA
`
`722
`
`724
`
`OPTIONALLY TRANSMIT HINTS TO ANOTHER
`DIGITAL PROCESSING SYSTEM
`
`726
`
`FIG. 10
`
`Page 00012
`
`Page 00012
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 11 of 14
`
`US 6,717,952 B2
`
`RECEIVE, AT A RECEIVING SYSTEM,
`MEDIA DATA TRANSMITTED TO THE
`
`RECEIVING SYSTEM IN ACCORDANCE
`
`WITH "HINTS"
`
`
`
`PRESENT, AT THE RECEIVING SYSTEM,
`A MEDIA OBJECT REPRESENTED BY
`
`THE MEDIA DATA
`
`730
`
`732
`
`OPTIONALLY STORE THE MEDIA DATA
`AS A MEDIA FILE
`
`734
`
`OPTIONALLY REASSEMBLE THE STORED
`MEDIA FILE
`
`736
`
`FIG. 11
`
`Page 00013
`
`Page 00013
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 12 of 14
`
`US 6,717,952 B2
`
`742
`
`744
`
`746
`
`GENERATOR
`OS
`
`HINT CREATION
`ROUT|NE(S)
`
`CREATED HINTS
`
`J
`
`762
`
`SERVER 03
`
`764
`
`766
`
`MEDIA DATA
`
`HINTS
`
`NETWORK TRANSMISSION
`ROUT|NE(S)
`
`INFORMATION RELATING
`TO ONE OR MORE DATA
`COMMUNICATION PROTOCOLS
`AND/OR MEDIA FORMATS
`
`MACHINE-READABLE
`STORAGE MEDIUM
`E
`
`FIG. 1 2
`
`NETWORK TRANSMISSION
`
`ROUTlNE(S)
`
`MEDIA PACKETIZATION
`ROUTlNE(S), BASED
`ON HINTS
`
`MACHINE-READABLE
`STORAGE MEDIUM
`
`Z§Q
`
`FIG. 13
`
`Page 00014
`
`Page 00014
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 13 of 14
`
`US 6,717,952 B2
`
`RECEIVING SYSTEM
`os
`
`MEDIA PRESENTATION
`ROUT|NE(S)
`
`(OPTIONAL)
`MEDIA DATA
`
`NETWORK TRANSMISSION
`F{OUT|NE(S)
`
`OPHONAL
`,I,,ED,A DATA
`FIE-ASSEMBLY ROUTINE
`
`MACHINE-FIEADABLE
`STORAGE MEDIUM
`7_*3°
`
`FIG. 14
`
`Page 00015
`
`Page 00015
`
`
`
`U.S. Patent
`
`Apr. 6, 2004
`
`Sheet 14 of 14
`
`US 6,717,952 B2
`
`806
`
`304
`
`HINT PACKET
`
`MEDIA DATA PACKET
`
` DATA STORAGE AND/OR
`
`COMMUNICATION MEDIUM
`
`E
`
`FIG. 15
`
`Page 00016
`
`Page 00016
`
`
`
`US 6,717,952 B2
`
`1
`METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`This application is a continuation of U.S. patent appli-
`cation Ser. No. 09/651,009, now U.S. Pat. No. 6,512,778,
`filed on Aug. 29, 2000, which is a continuation of U.S. patent
`application Ser. No. 09/140,173, now U.S. Pat. No. 6,134,
`243, filed on Aug. 25, 1998, which is a continuation-in-part
`of U.S. Patent Application 60/071,566, filed on Jan. 15,
`1998.
`
`FIELD OF THE INVENTION
`
`The present invention relates to methods and apparatuses
`for preparing time related sequences of media data for
`transmission, and more particularly to packetized transmis-
`sion of such media data.
`
`INTRODUCTION AND BACKGROUND
`
`There are various different file structures used today to
`store time-based media: audio formats such as AIFF, video
`formats such as AVI, and streaming formats such as Real-
`Media. One reason that such file structures are different is
`
`their different focus and applicability. Some of these formats
`are sufficiently relatively widely accepted, broad in their
`application, and somewhat simple to implement, and thus,
`may be used not only for content delivery but also as
`interchange formats. Foremost among these general formats
`is the QuickTime file format. It is used today in the majority
`of web sites serving time-based data;
`in the majority of
`authoring environments, including professional ones; and on
`the majority of multimedia CDROM titles.
`The QuickTime media layer supports the efficient display
`and management of general multimedia data, with an
`emphasis on time-based material (video, audio, etc.). The
`media layer uses the QuickTime file format as the storage
`and interchange format for media information. The archi-
`tectural capabilities of the layer are generally broader than
`the existing implementations, and the file format is capable
`of representing more information than is currently
`demanded by the existing QuickTime implementations.
`In contrast to formats such as AVI, which were generally
`designed to support local random access of synchronized
`media, QuickTime allows systems to manage the data,
`relationships and timing of a general multimedia presenta-
`tion. In particular, the QuickTime file format has structures
`to represent the temporal behavior of general time-based
`streams, a concept which covers the time-based emission of
`network packets, as well as the time-based local presentation
`of multimedia data.
`
`The existing QuickTime file format is publicly described
`by Apple Computer
`in the May 1996 File format
`specification, which may be found at the QuickTime site,
`<http://.www.apple.com/quicktime>.
`One aspect of the QuickTime file format is the concept
`that the physical structure of media data (the layout in disk
`records) is independent of, and described by, a logical
`structure for the file. The file is fully described by a set of
`“movie” meta-data. This meta-data provides declarative,
`structural and temporal information about the actual media
`data.
`
`The media data may be in the same file as the description
`data, (the “movie” meta-data), or in other file(s). A movie
`structured into one file is commonly called “flat”, and is
`self-contained. Non-flat movies can be structured to refer-
`ence some, or all, of the media data in other files.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`As such, the format is generally suited for optimization in
`different applications. For example, when editing
`(compositing), data need not be rewritten as edits are applied
`and media is re-ordered; the meta-data file may be extended
`and temporal mapping information adjusted. When edits are
`complete, the relevant media data and meta-data may be
`rewritten into a single, interleaved, and optimized file for
`local or network access. Both the structured and the opti-
`mized files are valid QuickTime files, and both may be
`inspected, played, and reworked.
`The use of structured (“non-flat”) files enables the same
`basic media data to be used and re-used in any number of
`presentations. This same advantage applies when serving, as
`will be seen below.
`
`In both editing and serving, this also permits a number of
`other files to be treated as part of a movie without copying
`the media data. Thus editing and serving may be done
`directly from files such as Sun Microsystem’s “au” audio
`format or the AVI video format, greatly extending the utility
`of these formats.
`
`The QuickTime file is divided into a set of objects, called
`atoms. Each object starts with an atom header, which
`declares its size and type:
`
`class Atom {
`
`}
`
`int(32)
`char
`byte
`
`size;
`type[4];
`contents[ ];
`
`The size is in bytes, including the size and type header
`fields. The type field is four characters (usually printable), to
`permit easy documentation and identification. The data in an
`object after the type field may be fields, a sequence of
`contained objects, or both.
`A file therefore is simply a sequence of objects:
`
`class File {
`Atom[ ];
`
`}
`
`The two important top-level objects are the media-data
`(mdat) and the meta-data (moov).
`The media-data object(s) contain the actual media (for
`example, sequences of sound samples). Their format is not
`constrained by the file format; they are not usually objects.
`Their format
`is described in the meta-data, not by any
`declarations physically contiguous with them. So,
`for
`example,
`in a movie consisting solely of motion-JPEG,
`JPEG frames are stored contiguously in the media data with
`no intervening extra headers. The media data within the
`media data objects is logically divided into chunks;
`however, there are no explicit chunk markers within the
`media data.
`
`When the QuickTime file references media data in other
`files, it is not required that these ‘secondary’ files be for-
`matted according to the QuickTime specification, since such
`media data files may be formatted as if they were the
`contents of a media obj ect. Since the QuickTime format does
`not necessarily require any headers or other information
`physically contiguous with the media data, it is possible for
`the media data to be files which contain ‘foreign’ headers
`(e.g. UNIX “.au” files, or AVI files) and for the QuickTime
`
`Page 00017
`
`Page 00017
`
`
`
`US 6,717,952 B2
`
`3
`meta-data to contain the appropriate declarative information
`and reference the media data in the ‘foreign’ file. In this way
`the QuickTime file format can be used to update, without
`copying, existing bodies of material in disparate formats.
`The QuickTime file format is both an established format and
`is able to work with, include, and thereby bring forward,
`other established formats.
`
`Free space (e.g. deleted by an editing operation) can also
`be described by an object. Software reading a file that
`includes free space objects should ignore such free space
`objects, as well as objects at any level which it does not
`understand. This permits extension of the file at virtually any
`level by introducing new objects.
`The primary meta-data is the movie object. A QuickTime
`file has exactly one movie object which is typically at the
`beginning or end of the file, to permit its easy location:
`
`class Movie {
`int(32)
`char
`MovieHeader
`contents
`
`size;
`type[4] = ‘moov’;
`mh;
`Atom[ ];
`
`The movie header provides basic information about the
`overall presentation (its creation date, overall timescale, and
`so on). In the sequence of contained objects there is typically
`at least one track, which describes temporally presented
`data.
`
`class Track {
`int(32)
`char
`TrackHeader
`contents
`
`size;
`type[4] = ‘trak’;
`th;
`Atom[ ];
`
`The track header provides relatively basic information
`about
`the track (its ID,
`timescale, and so on). Objects
`contained in the track might be references to other tracks
`(e. g. for complex compositing), or edit lists. In this sequence
`of contained objects there may be a media object, which
`describes the media which is presented when the track is
`played.
`The media object contains declarations relating to the
`presentation required by the track (e.g. that it is sampled
`audio, or MIDI, or orientation information for a 3Dscene).
`The type of track is declared by its handler:
`
`class handler {
`int(32)
`char
`int(8)
`bit(24)
`char
`char
`audio
`char
`bit(32)
`bit(32)
`string
`
`}
`
`size;
`type[4] = ‘hdlr’;
`version;
`flags;
`handlertype[4];
`handlersubtype[4]
`
`manufacturer[4];
`handlerfiags;
`handlerfiagsmask;
`componentname;
`
`-- mhlr for media handlers
`-- vide for video, soun for
`
`Within the media information there is likewise a handler
`
`declaration for the data handler (which fetches media data),
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`and a data information declaration, which defines which files
`contain the media data for the associated track. By using this
`declaration, movies may be built which span several files.
`At the lowest level, a sample table is used which relates
`the temporal aspect of the track to the data stored in the file:
`
`class sampletable {
`size;
`int(32)
`type[4] = ‘stbl’;
`char
`sampledescription
`timetosample
`syncsampletable
`sampletochunk
`samplesize
`chunkoffset
`shadowsync
`
`sd;
`tts;
`syncs,
`stoc;
`ssize;
`coffset,
`ssync,
`
`The sample description contains information about the
`media (e.g.
`the compression formats used in video). The
`time-to-sample table relates time in the track, to the sample
`(by index) which should be displayed at that time. The sync
`sample table declares which of these are sync (key) samples,
`not dependent on other samples.
`The sample-to-chunk object declares how to find the
`media data for a given sample, and its description given its
`index:
`
`class sampletochunk {
`int(32)
`char
`int(8)
`bits(24)
`int(32)
`for (int i=0;
`int(32)
`int(32)
`int(32)
`
`}
`
`size;
`type[4] = ‘stsc’;
`version;
`flags;
`entrycount;
`i<entrycount; i++) {
`firstchunk;
`samplesperchunk;
`sampledescriptionindex;
`
`The sample size table indicates the size of each sample.
`The chunkoffset table indicates the offset into the containing
`file of the start of each chunk.
`
`Walking the above-described structure to find the appro-
`priate data to display for
`a given time is fairly
`straightforward, generally involving indexing and adding.
`Using the sync table, it is also possible to back-up to the
`preceding sync sample, and roll forward ‘silently’ accumu-
`lating deltas to a desired starting point.
`FIG. 1 shows the structure of a simple movie with one
`track. Asimilar diagram may be found in the QuickTime file
`format documentation, along with a detailed description of
`the fields of the various objects. QuickTime atoms (objects)
`are shown here with their type in a grey box, and a
`descriptive name above. This movie contains a single video
`track. The frames of video are in the same file, in a single
`chunk of data. It should be noted that the ‘chunk’ is a logical
`construct only;
`it
`is not an object. Inside the chunk are
`frames of video, typically stored in their native form. There
`are no required headers or fields in the video frames them-
`selves.
`
`FIG. 2 is a diagram of a self-contained file with both an
`audio and a video track. Fewer of the atoms are shown here,
`for brevity; the pointers from the tracks into the media data
`are, of course, the usual sample table declarations, which
`include timing information.
`
`Page 00018
`
`Page 00018
`
`
`
`US 6,717,952 B2
`
`5
`The QuickTime file format has a number of advantages,
`including:
`1) Scalability for size and bit-rates. The meta data is
`flexible, yet compact. This makes it suitable for small
`downloaded movies (e.g. on the Internet) as well as
`providing the basis for a number of high-end editing
`systems.
`2) Physical structure is independent of the logical and
`temporal structure. This makes it possible to optimize
`the physical structure differently depending on the use
`the file will have. In particular, it means that a single file
`format is suitable for authoring and editing; download-
`ing or placing on CDROMs; and for streaming.
`3) The file format has proven capable of handling a very
`broad variety of codec types and track types, including
`many not known at the time the format was designed.
`This proven ability to evolve in an upwards-compatible
`fashion is fundamental
`to the success of a storage
`format.
`
`Scalable, or layered, codecs can be handled in a number
`of ways in the QuickTime file format. For a streaming
`protocol which supports scalability,
`the samples may be
`tagged with the layer or bandwidth threshold to be met for
`transmitting the samples.
`Tracks which form a set of alternatives (e.g. different
`natural language sound tracks) can be tagged so that only
`one is selected for playback. The same structure can be used
`to select alternatives for streaming (e.g.
`for
`language
`selection). This capability is described in further detail in the
`QuickTime file format.
`When QuickTime displays a movie or track, the appro-
`priate media handler accesses the media data for a particular
`time. The media handler must correctly interpret the data
`stream to retrieve the requested data. For example, with
`respect to video media, the media handler typically traverses
`several atoms to find the location and size of a sample for a
`given media time. The media handler may perform the
`following:
`1. Determine the time in the media time coordinate
`system.
`2. Examine the time-to-sample atom to determine the
`sample number that contains the data for the specified
`time.
`
`3. Scan the sample-to-chunk atom to discover which
`chunk contains the sample in question.
`4. Extract the offset to the chunk from the chunk offset
`atom.
`
`5. Find the offset within the chunk and the sample’s size
`by using the sample size atom.
`It is often desirable to transmit a QuickTime file or other
`types of time related sequences of media data over a data
`communication medium, which may be associated with a
`computer network (e.g.
`the Internet). In many computer
`networks, the data which is transmitted into the network
`should generally be in a packet form. Normally, time related
`sequences of media data are not in the proper packetized
`format for transmission over a network. For example, media
`data files in the QuickTime format are not in a packetized
`format. Thus, there exists a need to collect the data, some-
`times referred to as streaming data, into packets for trans-
`mission over a network.
`
`One prior approach to address the problem of transmitting
`time related sequences of media data over a network is to
`send the media file over the network using a network or
`transmission protocol, such as the Hypertext Transfer Pro-
`tocol (HTTP). Thus, the media file itself is sent from one
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`computer system over the network to another computer
`system. However, there may be no desire to retain the media
`file at the receiving computing system. That is, when the
`media file is received and viewed or listened to at
`the
`
`receiving computer system, there may be no desire by the
`user of that receiving computer system to store a copy of the
`file, for example, if the receiving computing system is a
`network computer or a computer with low storage capacity.
`Another alternative approach to solving the problem of
`how to collect data for transmission by packets over a
`network is to prepare a file which contains the network
`protocol data units in the file for a particular transmission
`protocol.
`In a sense, such a file may be considered a
`packetized file which is stored in essentially the same format
`as it will be transmitted according to the particular trans-
`mission protocol. Performing this operation generally
`involves storing the file in a packetized form for a particular
`network protocol at a particular data transmission rate and a
`particular media file format. Thus, for each different trans-
`mission protocol at a particular data transmission rate, the
`file will essentially be replicated in its packetized form. The
`fixed form of such files may restrict
`their applicability/
`compatibility and make it difficult to view such files locally.
`Thus, such an approach may greatly increase storage
`requirements in attempting to provide the file in various
`transmission protocols at various different data transmission
`rates. Moreover, each packetized file generated according to
`this alternative prior approach is generally limited to a
`particular media file format, and thus, other media file
`formats for the same media object (e.g. a digital movie) are
`typically packetized and stored on the sending computer
`system.
`Yet another approach to solving the problem of how to
`stream time related sequences of media data is to perform
`the packetization of the media data when required on the
`transmitting system according to the particular transmission
`protocol which is desired. This processing requires, in many
`cases, a relatively considerable amount of time, and thus,
`may slow the performance of the transmitting system.
`Thus, it is desirable to provide an improved method and
`apparatus for transmitting time related sequences of media
`data.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides methods and apparatuses
`for processing media data for transmission in a data com-
`munication medium. In one embodiment, a set of data
`indicates how to transmit a time related sequence of media
`data according to a transmission protocol. The set of data,
`according to one embodiment,
`includes a time related
`sequence of data which is associated with the time related
`sequence of media data. According to one aspect of the
`invention,
`the set of data may be utilized by a digital
`processing system to transmit the time related sequence of
`media data (e.g., by packets generated according to the
`transmission protocol and the set of data).
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 shows an example of the structure of a simple
`movie with one track in the prior art.
`FIG. 2 is an example of a self-contained movie file of the
`prior art.
`FIG. 3 is a flowchart showing one example of a method
`according to the present invention.
`FIG. 4 shows an example of a hint track of the present
`invention.
`
`Page 00019
`
`Page 00019
`
`
`
`US 6,717,952 B2
`
`7
`FIG. 5 shows another example of a hint track of the
`present invention.
`FIG. 6 is a diagram of a network of computer systems in
`which media data may be exchanged and/or processed,
`according to one embodiment of the present invention.
`FIG. 7 is a block diagram of a digital processing system
`which may be used in accordance with one embodiment of
`the present invention.
`FIG. 8 is a block diagram of a system that utilizes hints
`to transfer media data, according to one embodiment of the
`invention.
`
`FIG. 9 is a block diagram of a system that utilizes hints
`to transfer media data, according to one embodiment of the
`invention.
`
`FIG. 10 is a flow diagram illustrating a method for
`generating hints for providing media data transmission,
`according to one embodiment of the invention.
`FIG. 11 is a flow diagram illustrating a method of pro-
`cessing media data received by a receiving system in accor-
`dance with hints, according to one embodiment of the
`invention.
`
`FIG. 12 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a generator, according to one embodiment of
`the invention.
`
`FIG. 13 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a server, according to one embodiment of the
`invention.
`
`FIG. 14 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a receiving system or other digital process-
`ing system, according to one embodiment of the invention.
`FIG. 15 is a diagram of a data storage and/or communi-
`cation medium having stored/transported thereon media and
`hint information, according to one embodiment of the inven-
`tion.
`
`DETAILED DESCRIPTION
`
`The present invention provides methods and apparatuses
`for allowing the transmission, and particularly the pack-
`etized transmission of time related sequences of media data,
`which may include, for example, video, audio, video and
`audio, etc., over a communication media, such as in a
`computer network.
`In one embodiment of the present invention, a digital
`processing system creates a set of data for indicating how to
`transmit a time related sequence of media data according to
`a transmission protocol. Typically, this set of data is stored
`on a storage device coupled to the digital processing system.
`Further, this set of data is a time related sequence of data
`associated with the time related sequence of media data.
`The present invention may be implemented entirely in
`executable computer program instructions which are stored
`on a computer readable media or may be implemented in a
`combination of software and hardware, or
`in certain
`embodiments, entirely in hardware. Typically, a server com-
`puter system coupled to a network will create the set of data,
`which may be referred to as a hint track and will store this
`hint track in a storage device which is coupled to the server
`computer system. When a client computer system requests a
`presentation (e.g. a viewing or listening or viewing and
`listening) of a media data file, the server system uses the hint
`track to determine how to packetize the media data for
`transmission to the client computer system. It will be appre-
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`ciated that the present invention is generally applicable to
`time related sequences of media data, and that QuickTime is
`represented herein as one example of this general applica-
`bility. Thus, the invention should not necessarily be limited
`to QuickTime.
`FIG. 3 shows one example of a method according to the
`present invention. The method 300 shown in FIG. 3 begins
`in step 301, in which the media file format for the particular
`media data which is desired to be transmitted is determined.
`In step 303, the particular transmission protocol or protocols
`which are desired to be used is also determined. However,
`steps 301 and 303 are optional, for example, in the case
`where the same media file format is always transmitted
`using the same transmission protocol.
`In step 305, a digital processing system, such as a server
`computer system, creates and stores the hints for packetizing
`a time related sequence of media data in a media file.
`Alternatively, one computer system may create the hints and
`provide them to another system, such as a server computer
`system, which stores