`
`(12) United States Patent
`US 7,366,788 B2
`(10) Patent N0.:
`
` Jones et al. (45) Date of Patent: *Apr. 29, 2008
`
`
`(54) METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`(75)
`
`Inventors: Anne Jones, Redwood City, CA (US);
`Jay Geagan, San Jose, CA (US); Kevin
`L' Gong’ Sunnyvale, CA (Us); Alag“
`Pen/annals san Framsco’ .CA (Us);
`Dav1d W. Singer, San Franc1sco, CA
`(US)
`
`(73) Assignee: Apple Inc., Cupertino, CA (US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 733 days.
`This patent is subject to a terminal dis-
`claimer.
`
`EP
`
`35333123; 2
`4,688,214 A
`5:319:707 A
`5,448,568 A
`
`1
`t
`313;; gem”? et :1;
`8/1987 DZHVIl/igtltnelftlcall. e a.
`6/1994 Wasilewski et al.
`9/1995 Delpuch et al.
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`0 497 449 A2
`8/1992
`(Continued)
`OTHER PUBLICATIONS
`
`(21) Appl. N0.: 10/789,582
`
`(22)
`
`Filed:
`
`Feb. 26, 2004
`
`(65)
`
`Prior Publication Data
`
`US 2004/0205224 A1
`
`Oct. 14, 2004
`
`Aaron E. Walsh,“Multimedia t0 the MACS”, Dr. Dobb’s Journal,
`Jul. 1992, p. 76, 78-80.
`
`(Continued)
`
`Primary ExamineriKrisna Lim
`(74) Attorney, Agent, or FirmiBlakely, Sokoloif, Taylor &
`Zafman LLP
`
`Related US. Application Data
`
`(57)
`
`ABSTRACT
`
`(63) Continuation of application No. 10/177,119, filed on
`Jun. 21, 2002, now Pat. No. 6,714,984, which is a
`continuation of application No. 09/139,378, filed on
`Aug. 25, 1998, now Pat. No. 6,453,355.
`
`(60) Provisional application No. 60/071,566, filed on Jan.
`15, 1998.
`
`(51)
`
`Int. Cl.
`(2006.01)
`G06F 15/16
`(52) US. Cl.
`....................... 709/230; 709/231; 370/394
`(58) Field of Classification Search ........ 709/2307231;
`345/327, 259; 370/394
`See application file for complete search history.
`
`Methods and apparatuses for processing media data trans-
`mitted in a data communication medium. A digital process-
`ing system is provided with a time related sequence of media
`data provided to the digital processing system based on a set
`of data, wherein the set of data indicates a method to
`transmit the time related sequence of media data according
`to a transmission protocol. The set of data, itself, is a time
`related sequence of data associated with the time related
`sequence of media data. The time related sequence of media
`data may be presented and/or stored by the digital process-
`ing system.
`
`59 Claims, 14 Drawing Sheets
`
`/ 300
`
`
`
`DETERMINE MEDIA FILE FORMAT
`
`DETERMINE DESIRED TRANSMISSION PROTOCOL(S)
`
`
`CREATE AND STORE "HINTS' FOR PACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA FILE(S) (PACKETIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`
`
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA FILE(S) AND HINTS TO A RECEIVING SYSTEM; DATA IS
`
`
`TRANSMITTED BY PACKETIZING THE MEDIA DATA ACCORDING TO
`THE HINTS
`
`309
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`
`311
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS 0F MEDIA DATA RECEIVED AT THE
`
`
`RECEIVING SYSTEM
`
`
`301
`
`303
`
`305
`
`307
`
`Apple Exhibit 4432
`
`Apple V. SightSound Technologies
`CBM2013-00023
`
`Page 00001
`
`Apple Exhibit 4432
`Apple v. SightSound Technologies
`CBM2013-00023
`Page 00001
`
`
`
`US 7,366,788 B2
`
`Page 2
`
`US. PATENT DOCUMENTS
`
`5,497,373 A
`5,544,198 A
`5,623,490 A
`5,655,117 A
`5,659,539 A
`5,694,334 A
`5,774,666 A
`5,778,187 A
`5,784,277 A
`5,799,150 A
`5,802,294 A
`5,818,441 A
`5,826,024 A
`5,838,678 A
`5,864,682 A
`5,915,094 A
`5,928,330 A
`5,956,729 A
`5,966,120 A
`5,987,501 A
`5,987,509 A
`6,055,246 A
`6,064,771 A
`
`3/ 1996 Hulen et al~
`8/ 1996 Saalfrank et al~
`4/1997 Richter et al.
`8/1997 Goldberg et al.
`8/ 1997 Porter et al~
`12/1997 Donahue et al.
`6/1998 Portuesi
`7/1998 Monteiro et al.
`7/1998 Meyer
`8/1998 Hamilton et al.
`9/1998 Ludwig et a1.
`10/1998 Throckmorton et al.
`10/ 1998 Higashimura et al.
`11/1998 Davis et al.
`1/1999 Porter et a1.
`6/1999 Kouloheris et 31~
`7/1999 Goetz 6t 31.
`9/1999 Goetz 6t 31.
`10/1999 Arazi et al.
`11/1999 Hamilton et al.
`11/1999 Portuesi
`4/2000 Jones
`5/2000 Migdal et al.
`
`...... 709/214
`
`6,098,188 A *
`6,112,226 A
`6,157,674 A
`6,175,871 B1
`6,327,418 B1
`
`8/2000 Kalmanek et a1.
`8/2000 Weaver et al.
`12/2000 Oda et a1.
`1/2001 Schuster et a1.
`12/2001 Barton
`
`.......... 714/746
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`W0
`
`0 702 309 A1
`1 458 196 A2
`W0 02/054284
`
`3/1996
`9/2004
`7/2002
`
`OTHER PUBLICATIONS
`
`Paul England, Robert Allen, Ron Underwood, “RAVE: Real-Time
`Services for the Web”, Computer Networks and ISDN Systems, pp.
`1547-1558.
`Susie J. Wee and John G. Apostolopoulos, “Secure Scalable Stream-
`ing Enabling Transcoding Without Decryption”, Proceedings 2002
`International Conference On Image Processing, ICIP, Oct. 7, 2001,
`vol. 1 of 3, Conf. 8, pp. 437-440, IEEE, USA.
`International Search Report, PCTflJS2006/028275, Dec. 18, 2006,
`11 pages.
`
`* cited by examiner
`
`Page 00002
`
`Page 00002
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 1 of 14
`
`US 7,366,788 B2
`
`movie header
`
`_rckheader
`
`media data
`
`D
`
`Cl D
`
`D E
`
`Cl C
`
`U
`
`
`
`
`
`
`
`mdat chunk Shame: Shame: Shame: Shame:
`D
`D U
`D D
`U D
`U
`
`- - -
`
`FIG. 1
`
`Page 00003
`
`Page 00003
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 2 of 14
`
`US 7,366,788 B2
`
`movie header
`
`Page 00004
`
`Page 00004
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 3 of 14
`
`US 7,366,788 B2
`
`[- 300
`
`DETERMINE MEDIA FILE FORMAT
`
`DETERMINE DESIRED TRANSMISSION PROTOCOL(S)
`
`CREATE AND STORE "HINTS" FOR PACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA FILE(S) (PACKETIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`
`
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA FILE(S) AND HINTS TO A RECEIVING SYSTEM; DATA IS
`TRANSMITTED BY PACKETIZING THE MEDIA DATA ACCORDING TO
`
`THE HINTS
`
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`
`
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`RECEIVING SYSTEM
`
`
`
`
`301
`
`303
`
`305
`
`307
`
`309
`
`311
`
`FIG. 3
`
`Page 00005
`
`Page 00005
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 4 of 14
`
`US 7,366,788 B2
`
`
`_-_V-i—PPackeihi PPackeihi STE? PPackeihi
`
`
`
`MEMI-uint_er
`
`
`Mhde-'nint_er Eheadel-uin_ter
`
`
`
`l@ram-E
`sampe
`sample
`sample
`
`-405
`
`
`
`
`
`FIG. 4
`
`Page 00006
`
`Page 00006
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 5 of 14
`
`US 7,366,788 B2
`
`
`
`SImpIe MOVle—Vldeo only
`
`
`track
`
`(video)
`
`
`
`
`HintMovi- Videometa-daEll'Pmeta-dataam:-.
`
`-—-T_-—-
`
`
`RTPPackeihinilsampleI
`
`
`'aRTPPckerinhfiitl WWImle sampel
`
`headel__'nin_ierI
`
`
`headlr.uinie_iI__ headeisa'i_nt__e_rl
`
`
`
`
`
`track _——
`mp hint
`
`I—__-
`
`
`
`Page 00007
`
`Page 00007
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 6 of 14
`
`US 7,366,788 B2
`
`602
`
`
`CUENT
`COMPUTER
`
`
`SYSTEM
`
`
`
`
`CUENT
`
`COMPUTER
`
`SYSTEM
`
`INTERNET
`
`\NEB
`
`SERVER
`
`SYSTEM
`
`61
`
`NERNORK
`
`INTERFACE
`
`
`
`CUENT
`
`COMPUTER
`SYSTEM
`
`CUENT
`
`
`
`
`COMPUTER
`
`SYSTEM
`
`
`
`
`618
`620
`
`FIG. 6
`
`Page 00008
`
`Page 00008
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 7 of 14
`
`US 7,366,788 B2
`
`652
`
`PROCESSOR
`
`MEMORY
`
`654
`
`656
`
`658
`
`
`
`
`
`
`
`DISPLAY
`CONTROLLER
`
`MASS
`MEMORY
`
`I/O
`CONTROLLER
`
`664
`
`662
`
`660
`
`DISPLAY
`
`I/O
`DEVICES
`
`666
`
`DIGITAL PROCESSING
`
`
`
`SYSTEM
`
`
`652
`
`
`
`MODEM
`
`OR
`
` 668
`
`NETWORK
`
`
`INTERFACE
`
`
`FIG. 7
`
`Page 00009
`
`Page 00009
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 8 of 14
`
`US 7,366,788 B2
`
`
`
`MEDIA
`
`
`PROCESSING
`
`
`UNIT
`
`
`
`
`HINT GENERATION
`
`AND PROCESSING UNIT
`
`690
`
`MEDIA PROCESSING
`
`UNIT
`
` 688
`
`
`
`
` 684
`
`
`CLIENT DATA
`PROCESSING SYSTEM
`683
`
`LINK 686
`
`
`692
`DATA
`
`
`COMMUN'CAT'ON
`DATA COMMUNICATION
`
` SERVER ggg
`
`UNIT
`
`SYSTEM 680 /
`
`FIG. 8
`
`Page 00010
`
`Page 00010
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 9 of 14
`
`US 7,366,788 B2
`
`702
`
`HINT PROCESSING
`
`UNIT
`
`704
`
`MEDIA PROCESSING
`UNIT
`
`706
`
`DATA COMMUNICATION
`UNIT
`
`
`
`
`
`
`
`
`MEDIA
`PROCESSING
`
`
`UNIT
`
`
`
`
`PROCESSING SYSTEM
`682
`
`684
` CLIENT DATA
`
`
`
`LINK 686
`
`
`
`DATA
`COMMUNICATION
`
`
`SYSTEM 696
`
`/
`
`DATA
`
`COMMUNICATION
`LINK 708
`
`SERVER m
`
`
`
`
`
`
`712
`
`
`
`
`HINT GENERATION
`
`UNIT
`
`GENERATOR _7_19
`
`
`
`FIG. 9
`
`Page 00011
`
`Page 00011
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 10 of 14
`
`US 7,366,788 B2
`
`DETERMINE MEDIA FORMAT (IF MORE THAN
`ONE FORMAT IS/WILL BE USED)
`
`720
`
`DETERMINE DESIRED DATA COMMUNICATION
`PROTOCOL(S) (IF MORE THAN ONE PROTOCOL
`ISNVILL BE USED)
`
`
`
`BASED ON THE MEDIA FORMAT AND DESIRED
`DATA COMMUNICATION PROTOCOL(S), CREATE
`AND STORE HINTS FOR PACKETIZING A
`
`TIME RELATED SEQUENCE OF MEDIA DATA
`
`722
`
`724
`
`OPTIONALLY TRANSMIT HINTS TO ANOTHER
`DIGITAL PROCESSING SYSTEM
`
`726
`
`FIG. 10
`
`Page 00012
`
`Page 00012
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 11 of 14
`
`US 7,366,788 B2
`
`RECEIVE, AT A RECEIVING SYSTEM,
`MEDIA DATA TRANSMITTED TO THE
`RECEIVING SYSTEM IN ACCORDANCE
`
`WITH "HINTS"
`
`730
`
`PRESENT, AT THE RECEIVING SYSTEM,
`A MEDIA OBJECT REPRESENTED BY
`
`THE MEDIA DATA
`
`732
`
`OPTIONALLY STORE THE MEDIA DATA
`AS A MEDIA FILE
`
`734
`
`OPTIONALLY REASSEMBLE THE STORED
`MEDIA FILE
`
`736
`
`FIG. 11
`
`Page 00013
`
`Page 00013
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 12 of 14
`
`US 7,366,788 B2
`
`GENERATOR
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`OS
`
`
`INFORMATION RELATING
`
`
`
`HINT CREATION
`ROUTINE(S)
`
`
`
`
`
`TO ONE OR MORE DATA
`COMMUNICATION PROTOCOLS
`
`AND/OR MEDIA FORMATS
`
`
`CREATED HINTS
`
`746
`
`
`
`
`MACHINE-READABLE
`STORAGE MEDIUM
`740
`
`
`m
`
`ROUTINE(S)
`
`MEDIA PACKETIZATION
`ROUTINE(S), BASED
`ON HINTS
`
`MACHINE-READABLE
`
`STORAGE MEDIUM
`
`MEDIA DATA
`
`FIG. 13
`
`Page 00014
`
`Page 00014
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 13 of 14
`
`US 7,366,788 B2
`
`
`
`
`
`(OPTIONAL)
`MEDIA DATA
`
`FIE-ASSEMBLY ROUTINE
`
`
`
`_
`
`772
`
`RECEIVING SYSTEM
`
`OS
`
`(OPTIONAL)
`MEDIA DATA
`
`774
`
`776
`
`
`
`
`
`
`778
`
`MEDIA PRESENTATION
`ROUTINE(S)
`
`782
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`MACHINE-READABLE
`
`STORAG7I§0MEOIUM
`
`FIG. 14
`
`Page 00015
`
`Page 00015
`
`
`
`U.S. Patent
`
`Apr. 29, 2008
`
`Sheet 14 of 14
`
`US 7,366,788 B2
`
`HINT PACKET
`
`MEDIA DATA PACKET
`
`800
`
`DATA STORAGE AND/OR
`
`COMMUNICATION MEDIUM
`
`FIG. 15
`
`Page 00016
`
`Page 00016
`
`
`
`US 7,366,788 B2
`
`1
`METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`This application is a continuation application of Ser. No.
`10/177,119, now US. Pat. No. 6,714,984, filed Jun. 21,
`2002, which is a continuation of Ser. No. 09/139,378, now
`US. Pat. No. 6,453,355, filed Aug. 25, 1998, which is a
`provisional application of Ser. No. 60/071,566, filed Jan. 15,
`1998.
`
`FIELD OF THE INVENTION
`
`The present invention relates to methods and apparatuses
`for preparing time related sequences of media data for
`transmission, and more particularly to packetized transmis-
`sion of such media data.
`
`INTRODUCTION AND BACKGROUND
`
`There are various different file structures used today to
`store time-based media: audio formats such as AIFF, video
`formats such as AVI, and streaming formats such as Real-
`Media. One reason that such file structures are different is
`
`their different focus and applicability. Some of these formats
`are sufficiently relatively widely accepted, broad in their
`application, and somewhat simple to implement, and thus,
`may be used not only for content delivery but also as
`interchange formats. Foremost among these general formats
`is the QuickTime file format. It is used today in the majority
`of web sites serving time-based data;
`in the majority of
`authoring environments, including professional ones; and on
`the majority of multimedia CDROM titles.
`The QuickTime media layer supports the efficient display
`and management of general multimedia data, with an
`emphasis on time-based material (video, audio, etc.). The
`media layer uses the QuickTime file format as the storage
`and interchange format for media information. The archi-
`tectural capabilities of the layer are generally broader than
`the existing implementations, and the file format is capable
`of
`representing more
`information than is
`currently
`demanded by the existing QuickTime implementations.
`In contrast to formats such as AVI, which were generally
`designed to support local random access of synchronized
`media, QuickTime allows systems to manage the data,
`relationships and timing of a general multimedia presenta-
`tion. In particular, the QuickTime file format has structures
`to represent the temporal behavior of general time-based
`streams, a concept which covers the time-based emission of
`network packets, as well as the time-based local presentation
`of multimedia data.
`
`The existing QuickTime file format is publicly described
`by Apple Computer in the May 1996 File format specifica-
`tion, which may be found at
`the QuickTime
`site,
`<www.apple.com/quicktime>.
`One aspect of the QuickTime file format is the concept
`that the physical structure of media data (the layout in disk
`records) is independent of, and described by, a logical
`structure for the file. The file is fully described by a set of
`“movie” meta-data. This meta-data provides declarative,
`structural and temporal information about the actual media
`data.
`
`The media data may be in the same file as the description
`data, (the “movie” meta-data), or in other file(s). A movie
`structured into one file is commonly called “flat”, and is
`self-contained. Non-flat movies can be structured to refer-
`ence some, or all, of the media data in other files.
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`As such, the format is generally suited for optimization in
`different applications. For example, when editing (compos-
`iting), data need not be rewritten as edits are applied and
`media is re-ordered; the meta-data file may be extended and
`temporal mapping information adjusted. When edits are
`complete, the relevant media data and meta-data may be
`rewritten into a single, interleaved, and optimized file for
`local or network access. Both the structured and the opti-
`mized files are valid QuickTime files, and both may be
`inspected, played, and reworked.
`The use of structured (“non-flat”) files enables the same
`basic media data to be used and re-used in any number of
`presentations. This same advantage applies when serving, as
`will be seen below.
`
`In both editing and serving, this also permits a number of
`other files to be treated as part of a movie without copying
`the media data. Thus editing and serving may be done
`directly from files such as Sun Microsystem’s “au” audio
`format or the AVI video format, greatly extending the utility
`of these formats.
`
`The QuickTime file is divided into a set of objects, called
`atoms. Each object starts with an atom header, which
`declares its size and type:
`
`class Atom {
`1mm)
`char
`byte
`
`}
`
`size;
`typ€[4l;
`contents [ ];
`
`The size is in bytes, including the size and type header
`fields. The type field is four characters (usually printable), to
`permit easy documentation and identification. The data in an
`object after the type field may be fields, a sequence of
`contained objects, or both.
`A file therefore is simply a sequence of objects:
`
`class File {
`Atom[ ];
`
`}
`
`The two important top-level objects are the media-data
`(mdat) and the meta-data (moov).
`The media-data object(s) contain the actual media (for
`example, sequences of sound samples). Their format is not
`constrained by the file format; they are not usually objects.
`Their format
`is described in the meta-data, not by any
`declarations physically contiguous with them. So,
`for
`example,
`in a movie consisting solely of motion-JPEG,
`JPEG frames are stored contiguously in the media data with
`no intervening extra headers. The media data within the
`media data objects is logically divided into chunks; how-
`ever, there are no explicit chunk markers within the media
`data.
`
`When the QuickTime file references media data in other
`files, it is not required that these ‘secondary’ files be for-
`matted according to the QuickTime specification, since such
`media data files may be formatted as if they were the
`contents of a media object. Since the QuickTime format does
`not necessarily require any headers or other information
`physically contiguous with the media data, it is possible for
`the media data to be files which contain ‘foreign’ headers
`(e.g. UNIX “.au” files, or AVI files) and for the QuickTime
`meta-data to contain the appropriate declarative information
`
`Page 00017
`
`Page 00017
`
`
`
`US 7,366,788 B2
`
`3
`and reference the media data in the ‘foreign’ file. In this way
`the QuickTime file format can be used to update, without
`copying, existing bodies of material in disparate formats.
`The QuickTime file format is both an established format and
`is able to work with, include, and thereby bring forward,
`other established formats.
`
`Free space (e.g. deleted by an editing operation) can also
`be described by an object. Software reading a file that
`includes free space objects should ignore such free space
`objects, as well as objects at any level which it does not
`understand. This permits extension of the file at Virtually any
`level by introducing new objects.
`The primary meta-data is the movie object. A QuickTime
`file has exactly one movie object which is typically at the
`beginning or end of the file, to permit its easy location:
`
`class Movie {
`int(32)
`char
`MovieHeader
`contents
`
`size;
`type[4] = ‘moov’;
`mh;
`Atom[ ];
`
`The movie header provides basic information about the
`overall presentation (its creation date, overall timescale, and
`so on). In the sequence of contained objects there is typically
`at least one track, which describes temporally presented
`data.
`
`class Track {
`int(32)
`char
`TrackHeader
`contents
`
`size;
`type[4] = ‘trak’;
`th;
`Atom[ ];
`
`The track header provides relatively basic information
`about
`the track (its ID,
`timescale, and so on). Objects
`contained in the track might be references to other tracks
`(e. g. for complex compositing), or edit lists. In this sequence
`of contained objects there may be a media object, which
`describes the media which is presented when the track is
`played.
`The media object contains declarations relating to the
`presentation required by the track (e.g. that it is sampled
`audio, or MIDI, or orientation information for a 3D scene).
`The type of track is declared by its handler:
`
`size;
`type[4] = ‘hdlr’;
`version;
`flags;
`-- mhlr for media handlers
`handlertype[4];
`handlersubtype[4] -- vide for video, soun for
`
`class handler {
`int(32)
`char
`int(8)
`bit(24)
`char
`char
`audio
`char
`bit(32)
`bit(32)
`string
`
`manufacturer[4] ;
`handlerfiags;
`handlerflagsmask;
`componentname;
`
`}
`
`Within the media information there is likewise a handler
`
`declaration for the data handler (which fetches media data),
`and a data information declaration, which defines which files
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`contain the media data for the associated track. By using this
`declaration, movies may be built which span several files.
`At the lowest level, a sample table is used which relates
`the temporal aspect of the track to the data stored in the file:
`
`‘stbl’;
`
`class sampletable {
`int(32)
`size;
`char
`type[4] =
`sampledescription sd;
`timetosample
`tts;
`syncsampletable
`syncs;
`sampletochunk
`stoc;
`samplesize
`ssize;
`chunkofiset
`coffset;
`shadowsync
`ssync;
`
`The sample description contains information about the
`media (e.g. the compression formats used in video). The
`time-to-sample table relates time in the track, to the sample
`(by index) which should be displayed at that time. The sync
`sample table declares which of these are sync (key) samples,
`not dependent on other samples.
`The sample-to-chunk object declares how to find the
`media data for a given sample, and its description given its
`index:
`
`‘stsc’;
`
`class sampletochunk {
`int(32)
`size;
`char
`type[4] =
`int(8)
`version;
`bits(24)
`flags;
`int(32)
`entrycount;
`for (int i=0; i<entrycount; i++) {
`int(32)
`firstchunk;
`int(32)
`samplesperchunk;
`int(32)
`sampledescriptionindex;
`
`The sample size table indicates the size of each sample.
`The chunkolfset table indicates the offset into the containing
`file of the start of each chunk.
`
`Walking the above-described structure to find the appro-
`priate data to display for a given time is fairly straightfor-
`ward, generally involving indexing and adding. Using the
`sync table, it is also possible to back-up to the preceding
`sync sample, and roll forward ‘silently’ accumulating deltas
`to a desired starting point.
`FIG. 1 shows the structure of a simple movie with one
`track. A similar diagram may be found in the QuickTime file
`format documentation, along with a detailed description of
`the fields of the various objects. QuickTime atoms (objects)
`are shown here with their type in a grey box, and a
`descriptive name above. This movie contains a single video
`track. The frames of video are in the same file, in a single
`chunk of data. It should be noted that the ‘chunk’ is a logical
`construct only;
`it
`is not an object. Inside the chunk are
`frames of video, typically stored in their native form. There
`are no required headers or fields in the video frames them-
`selves.
`
`FIG. 2 is a diagram of a self-contained file with both an
`audio and a video track. Fewer of the atoms are shown here,
`for brevity; the pointers from the tracks into the media data
`are, of course, the usual sample table declarations, which
`include timing information.
`
`Page 00018
`
`Page 00018
`
`
`
`US 7,366,788 B2
`
`5
`The QuickTime file format has a number of advantages,
`including:
`1) Scalability for size and bit-rates. The meta data is
`flexible, yet compact. This makes it suitable for small
`downloaded movies (e.g. on the Internet) as well as
`providing the basis for a number of high-end editing
`systems.
`2) Physical structure is independent of the logical and
`temporal structure. This makes it possible to optimize
`the physical structure differently depending on the use
`the file will have. In particular, it means that a single file
`format is suitable for authoring and editing; download-
`ing or placing on CDROMs; and for streaming.
`3) The file format has proven capable of handling a very
`broad variety of codec types and track types, including
`many not known at the time the format was designed.
`This proven ability to evolve in an upwards-compatible
`fashion is fundamental
`to the success of a storage
`format.
`
`Scalable, or layered, codecs can be handled in a number
`of ways in the QuickTime file format. For a streaming
`protocol which supports scalability, the samples may be
`tagged with the layer or bandwidth threshold to be met for
`transmitting the samples.
`Tracks which form a set of alternatives (e.g. different
`natural language sound tracks) can be tagged so that only
`one is selected for playback. The same structure can be used
`to select alternatives for streaming (e.g. for language selec-
`tion). This capability is described in further detail in the
`QuickTime file format.
`When QuickTime displays a movie or track, the appro-
`priate media handler accesses the media data for a particular
`time. The media handler must correctly interpret the data
`stream to retrieve the requested data. For example, with
`respect to video media, the media handler typically traverses
`several atoms to find the location and size of a sample for a
`given media time. The media handler may perform the
`following:
`1. Determine the time in the media time coordinate
`system.
`2. Examine the time-to-sample atom to determine the
`sample number that contains the data for the specified
`time.
`
`3. Scan the sample-to-chunk atom to discover which
`chunk contains the sample in question.
`4. Extract the offset to the chunk from the chunk offset
`atom.
`
`5. Find the offset within the chunk and the sample’s size
`by using the sample size atom.
`It is often desirable to transmit a QuickTime file or other
`types of time related sequences of media data over a data
`communication medium, which may be associated with a
`computer network (e.g. the Internet). In many computer
`networks, the data which is transmitted into the network
`should generally be in a packet form. Normally, time related
`sequences of media data are not in the proper packetized
`format for transmission over a network. For example, media
`data files in the QuickTime format are not in a packetized
`format. Thus, there exists a need to collect the data, some-
`times referred to as streaming data, into packets for trans-
`mission over a network.
`
`One prior approach to address the problem of transmitting
`time related sequences of media data over a network is to
`send the media file over the network using a network or
`transmission protocol, such as the Hypertext Transfer Pro-
`tocol (HTTP). Thus, the media file itself is sent from one
`computer system over the network to another computer
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`system. However, there may be no desire to retain the media
`file at the receiving computing system. That is, when the
`media file is received and viewed or listened to at the
`
`receiving computer system, there may be no desire by the
`user of that receiving computer system to store a copy of the
`file, for example, if the receiving computing system is a
`network computer or a computer with low storage capacity.
`Another alternative approach to solving the problem of
`how to collect data for transmission by packets over a
`network is to prepare a file which contains the network
`protocol data units in the file for a particular transmission
`protocol. In a sense, such a file may be considered a
`packetized file which is stored in essentially the same format
`as it will be transmitted according to the particular trans-
`mission protocol. Performing this operation generally
`involves storing the file in a packetized form for a particular
`network protocol at a particular data transmission rate and a
`particular media file format. Thus, for each different trans-
`mission protocol at a particular data transmission rate, the
`file will essentially be replicated in its packetized form. The
`fixed form of such files may restrict their applicability/
`compatibility and make it difficult to view such files locally.
`Thus,
`such an approach may greatly increase storage
`requirements in attempting to provide the file in various
`transmission protocols at various different data transmission
`rates. Moreover, each packetized file generated according to
`this alternative prior approach is generally limited to a
`particular media file format, and thus, other media file
`formats for the same media object (e.g. a digital movie) are
`typically packetized and stored on the sending computer
`system.
`Yet another approach to solving the problem of how to
`stream time related sequences of media data is to perform
`the packetization of the media data when required on the
`transmitting system according to the particular transmission
`protocol which is desired. This processing requires, in many
`cases, a relatively considerable amount of time, and thus,
`may slow the performance of the transmitting system.
`Thus, it is desirable to provide an improved method and
`apparatus for transmitting time related sequences of media
`data.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides methods and apparatuses
`for processing media data transmitted in a data communi-
`cation medium. In one embodiment, a digital processing
`system receives a time related sequence of media data
`provided to the digital processing system based on a set of
`data, wherein the set of data indicates a method to transmit
`the time related sequence of media data according to a
`transmission protocol. The set of data, itself, is a time related
`sequence of data associated with the time related sequence
`of media data. The time related sequence of media data may
`be presented and/or stored by the digital processing system.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 shows an example of the structure of a simple
`movie with one track in the prior art.
`FIG. 2 is an example of a self-contained movie file of the
`prior art.
`FIG. 3 is a flowchart showing one example of a method
`according to the present invention.
`FIG. 4 shows an example of a hint track of the present
`invention.
`
`Page 00019
`
`Page 00019
`
`
`
`US 7,366,788 B2
`
`7
`FIG. 5 shows another example of a hint track of the
`present invention.
`FIG. 6 is a diagram of a network of computer systems in
`which media data may be exchanged and/or processed,
`according to one embodiment of the present invention.
`FIG. 7 is a block diagram of a digital processing system
`which may be used in accordance with one embodiment of
`the present invention.
`FIG. 8 is a block diagram of a system that utilizes hints
`to transfer media data, according to one embodiment of the
`invention.
`
`FIG. 9 is a block diagram of a system that utilizes hints
`to transfer media data, according to one embodiment of the
`invention.
`
`FIG. 10 is a flow diagram illustrating a method for
`generating hints for providing media data transmission,
`according to one embodiment of the invention.
`FIG. 11 is a flow diagram illustrating a method of pro-
`cessing media data received by a receiving system in accor-
`dance with hints, according to one embodiment of the
`invention.
`
`FIG. 12 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a generator, according to one embodiment of
`the invention.
`
`FIG. 13 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a server, according to one embodiment of the
`invention.
`
`FIG. 14 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a receiving system or other digital process-
`ing system, according to one embodiment of the invention.
`FIG. 15 is a diagram of a data storage and/or communi-
`cation medium having stored/transported thereon media and
`hint information, according to one embodiment of the inven-
`tion.
`
`DETAILED DESCRIPTION
`
`The present invention provides methods and apparatuses
`for allowing the transmission, and particularly the pack-
`etized transmission of time related sequences of media data,
`which may include, for example, video, audio, video and
`audio, etc., over a communication media, such as in a
`computer network.
`In one embodiment of the present invention, a digital
`processing system creates a set of data for indicating how to
`transmit a time related sequence of media data according to
`a transmission protocol. Typically, this set of data is stored
`on a storage device coupled to the digital processing system.
`Further, this set of data is a time related sequence of data
`associated with the time related sequence of media data.
`The present invention may be implemented entirely in
`executable computer program instructions which are stored
`on a computer readable media or may be implemented in a
`combination of software and hardware, or in certain embodi-
`ments, entirely in hardware. Typically, a server computer
`system coupled to a network will create the set of data,
`which may be referred to as a hint track and will store this
`hint track in a storage device which is coupled to the server
`computer system. When a client computer system requests a
`presentation (e.g. a viewing or listening or viewing and
`listening) of a media data file, the server system uses the hint
`track to determine how to packetize the media data for
`transmission to the client computer system. It will be appre-
`ciated that the present invention is generally applicable to
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`time related sequences of media data, and that QuickTime is
`represented herein as one example of this general applica-
`bility. Thus, the invention should not necessarily be limited
`to QuickTime.
`FIG. 3 shows one example of a method according to the
`present invention. The method 300 shown in FIG. 3 begins
`in step 301, in which the media file format for the particular
`media data which is desired to be transmitted is determined.
`
`In step 303, the particular transmission protocol or protocols
`which are desired to be used is also determined. However,
`steps 301 and 303 are optional, for example, in the case
`where the same media file format is always transmitted
`using the same transmission protocol.
`In step 305, a digital processing system, such as a server
`computer system, creates and stores the hints for packetizing
`a time related sequence of media data in a media file.
`Alternatively, one computer system may create the hints and
`provide them to another system, such as a server computer
`system, which stores them for later use in a transmission
`process. The packetization allows the transmission over a
`network or communication media according to the desired
`transmission protocol which was determined in step 303. In
`one embodiment of the present
`invention,
`the hints are
`stored as a track of time related sequence of hints which
`refers to, but which in one embodiment,
`is separate from
`other tracks of media data. The track of hints,
`in one
`