(12) United States Patent
`US 7,747,765 B2
`Jones et al.
`(45) Date of Patent:
`*Jun. 29, 2010
`
`(10) Patent No.:
`
`US007747765B2
`
`(54) METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`(56)
`
`(75)
`
`Inventors: Anne Jones, Redwood City, CA (US);
`Jay Geagan, San Jose, CA (US); Kevin
`L. Gong, Sunnyvale, CA (US); Alagu
`Periyannan, San Francisco, CA (US);
`David W. Singer, San Francisco, CA
`(US)
`
`(73) Assignee: Apple Inc., Cupertino, CA (US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 315 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21) Appl.No.: 11/497,038
`
`(22)
`
`Filed:
`
`Jul. 31, 2006
`
`(65)
`
`Prior Publication Data
`
`US 2007/0067477 A1
`
`Mar. 22, 2007
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 10/789,582, filed on
`Feb. 26, 2004, now Pat. No. 7,366,788, which is a
`continuation of application No. 10/177,119, filed on
`Jun. 21, 2002, now Pat. No. 6,714,984, which is a
`continuation of application No. 09/139,378, filed on
`Aug. 25, 1998, now Pat. No. 6,453,355.
`
`(60) Provisional application No. 60/071,566, filed on Jan.
`15, 1998.
`
`(51)
`
`Int. Cl.
`(2006.01)
`G06F 15/16
`(52) U.S. Cl.
`........................ 709/230; 709/231; 370/394
`(58) Field of Classification Search ................. 709/230,
`709/231; 370/394; 345/327, 259
`See application file for complete search history.
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`3,873,777 A
`3,932,698 A
`4,688,214 A
`4,907,224 A
`5,251,209 A
`5,319,707 A
`5,365,272 A
`5,371,547 A
`5,404,469 A
`5,448,568 A
`5,497,373 A
`5,544,198 A
`5,574,939 A
`
`3/1975 Uehara et a1.
`1/1976 Yanagimachi et a1.
`8/1987 DeWitt et a1.
`3/1990 Scoles et a1.
`10/1993 Jurkevich et a1.
`6/1994 Wasilewski et a1.
`11/1994 Siracusa
`12/1994 Siracusa et a1.
`4/1995 Chung et a1.
`9/1995 Delpuch et a1.
`3/1996 Hulen et a1.
`8/1996 Saalfrank et a1.
`11/1996 Keckler et a1.
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`1298632
`4/1992
`
`CA
`
`(Continued)
`OTHER PUBLICATIONS
`
`Aaron E. Walsh,“Mu1timedja t0 the Macs”, Dr. Dobb’s Journal, Jul.
`1992, pp. 76, 78-80.
`
`(Continued)
`
`Primary ExamineriKrisna Lim
`(74) Attorney, Agent, or FirmiBlakely, Sokoloff, Taylor &
`Zafman LLP
`
`(57)
`
`ABSTRACT
`
`Methods and apparatuses for processing media data transmit-
`ted in a data communication medium. A digital processing
`system is provided with a time related sequence ofmedia data
`provided to the digital processing system based on a set of
`data, wherein the set of data indicates a method to transmit the
`time related sequence of media data according to a transmis-
`sion protocol. The set of data, itself, is a time related sequence
`of data associated with the time related sequence of media
`data. The time related sequence of media data may be pre-
`sented and/or stored by the digital processing system.
`
`59 Claims, 14 Drawing Sheets
`
`[300
`
`DETERMINE MEDIA FILE FORMAT
`
`
`
`DETERMINE DESIRED TRANSMISSION PROTOCOL(S)
`
`CREATE AND STORE 'HINTS' FOR PACKETIZING ATIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA FILE(S) (FACKETIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA FILE(S) AND HINTS TO A RECEIVING SYSTEM; DATA Is
`TRANSMITTED EV PACKETIZING THE MEDIA DATA ACCORDING To
`THE HINTS
`
`PRESENT. AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`RECEIVING SYSTEM
`
`301
`
`303
`
`305
`
`307
`
`309
`
`311
`
`Apple Exhibit 4440
`
`Apple V. SightSound Technologies
`CBM2013-00023
`
`Page 00001
`
`Apple Exhibit 4440
`Apple v. SightSound Technologies
`CBM2013-00023
`Page 00001
`
`

`

`US 7,747,765 B2
`
`Page 2
`
`US. PATENT DOCUMENTS
`
`>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
`
`5,623,490
`5,625,818
`5,655,117
`5,659,539
`5,689,509
`5,694,334
`5,768,535
`5,774,666
`5,778,187
`5,784,277
`5,799,150
`5,802,294
`5,818,441
`5,826,024
`5,838,678
`5,859,660
`5,864,682
`5,915,094
`5,928,330
`5,956,729
`5,966,120
`5,987,501
`5,987,509
`5,995,491
`6,055,246
`6,064,771
`6,098,188
`6,104,859
`6,112,226
`6,119,154
`6,134,243
`6,138,147
`6,157,674
`6,175,871
`6,175,872
`6,327,418
`6,438,172
`6,453,355
`6,512,778
`6,578,070
`6,714,984
`6,717,952
`6,744,763
`6,745,226
`6,829,648
`7,161,957
`
`4/1997
`4/1997
`8/1997
`8/1997
`11/1997
`12/1997
`6/1998
`6/1998
`7/1998
`7/1998
`8/1998
`9/1998
`10/1998
`10/1998
`11/1998
`1/1999
`1/1999
`6/1999
`7/1999
`9/1999
`10/1999
`11/1999
`11/1999
`11/1999
`4/2000
`5/2000
`8/2000
`8/2000
`8/2000
`9/2000
`10/2000
`10/2000
`12/2000
`1/2001
`1/2001
`12/2001
`8/2002
`9/2002
`1/2003
`6/2003
`3/2004
`4/2004
`6/2004
`6/2004
`12/2004
`1/2007
`
`Richter et a1.
`Zarmer et a1.
`Goldberg et al.
`Porter et al.
`Gaytan et al.
`Donahue et al.
`Chaddha et al.
`Portuesi
`Monteiro et a1.
`Meyer
`Hamilton et al.
`Ludwig et al.
`Throckmorton et al.
`Higashimura et al.
`Davis et a1.
`Perkins et a1.
`Porter et al.
`Kouloheris et al.
`Goetz et al.
`Goetz et al.
`Arazi et a1.
`Hamilton et al.
`Portuesi
`Richter et a1.
`Jones
`Migdal et al.
`Kalmanek et al.
`Yoshida et al.
`Weaver et al.
`Weaver et al.
`Jones et a1.
`Weaver et al.
`Oda et al.
`Schuster et al.
`Neumann et al.
`Barton
`Nakamura et al.
`Jones et a1.
`Jones et a1.
`Weaver et al.
`Jones et a1.
`Jones et a1.
`Jones et a1.
`Guedalia
`Jones et a1.
`Wang et a1.
`
`7,366,788 B2
`2002/0037037 A1
`2005/0195899 A1
`2005/0195900 A1
`2007/0022215 A1
`
`4/2008 Jones et al.
`3/2002 Van Der Schaar
`9/2005 Han
`9/2005 Han
`1/2007 Jones et a1.
`
`FOREIGN PATENT DOCUMENTS
`
`CA
`CA
`EP
`EP
`EP
`JP
`JP
`W0
`W0
`W0
`
`2 197 323
`2 387 254
`0 497 449 A2
`0 702 309 A1
`1 458 196 A2
`9101928 A
`9200158 A
`WO 97/22201
`WO 97/25817 A1
`W0 02/054284
`
`10/2001
`3/2003
`8/1992
`3/1996
`9/2004
`4/1997
`7/1997
`6/1997
`7/1997
`7/2002
`
`OTHER PUBLICATIONS
`
`Paul England, Robert Allen, Ron Underwood, “Rave: Real-Time
`Services for the Web”, Computer Networks and ISDN Systems, pp.
`1 547- 15 58.
`Susie J. Wee and John G. Apostolopoulos, “Secure Scalable Stream-
`ing Enabling Transcoding Without Decryption”, Proceedings 2002
`International Conference On Image Processing, ICIP, Oct. 7, 2001,
`vol. 1 of3, Conf. 8, pp. 437-440, IEEE, USA.
`International Search Report, PCT/US2006/028275, Dec. 18, 2006,
`11 pages.
`PCT International Search Report for PCT Int’l Application No. PCT/
`US99/00953 mailed Jul. 26, 1999.
`PCT International Search Report for PCT Int’l Application No. PCT/
`US99/00954 mailed Jul. 26, 1999.
`PCT International Search Report for PCT Int’l Application No. PCT/
`US99/00955 mailed Jul. 26, 1999.
`Song, Jun. “Synchronzing Feature ofMultimedia, ”Today’s Electron—
`ics, Jan. 18, 1997, pp. 30-31 in Chinese.
`Song, Jun. “Synchronzing Feature ofMultimedia, ”Today’s Electron—
`ics, Jan. 18, 1997, translated into English (pp. 1-6).
`Aaron E. Walsh,“Multimedia to the Macs”, Dr. Dobb’s Journal, Jul.
`1992, pp. 76, 78-80.
`Paul England, Robert Allen, Ron Underwood, “RAVE: Real-Time
`Services for the Web”, Computer Networks and ISDN Systems, pp.
`1547-1558, May 1996.
`Wee, Susie et al. “Secure Scalable Video Streaming for Wireless
`Networks.” IEEE International Conference on Acoustics, Speech,
`and Signal Processing, Salt Lake City, Utah, May 2001, 4 pages.
`
`Page 00002
`
`Page 00002
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 1 of 14
`
`US 7,747,765 B2
`
`
`
`mdat chunk Eframeé' Shame:
`D
`U D
`D E
`
`Eframeg - -
`III E
`D
`
`FIG. 1
`
`Page 00003
`
`media data
`
`C!
`
`D D
`
`D D
`
`U D
`
`D
`
`Page 00003
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 2 of 14
`
`US 7,747,765 B2
`
`
`
`
`_mvieheader
`II
`
`
`
`
`(video) track
`
`(audiO) track "—-
`
`
`
`
`Ikm___-__
`
`
`
`
`—..-._._-
`
`MMMM
`mflflfi
`I]D.
`I.I].
`
`
`
`DD
`
`FIG. 2
`
`Page 00004
`
`Page 00004
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 3 of 14
`
`US 7,747,765 B2
`
`r 300
`
`DETERMINE MEDIA FILE FORMAT
`
`DETERMINE DESIRED TRANSMISSION PROTOCOL(S)
`
`CREATE AND STORE "HINTS" FOR PACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA FILE(S) (PACKETIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`
`
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA FILE(S) AND HINTS TO A RECEIVING SYSTEM; DATA IS
`TRANSMITTED BY PACKETIZING THE MEDIA DATA ACCORDING TO
`THE HINTS
`
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`
` RECEIVING SYSTEM
`
`FIG. 3
`
`301
`
`303
`
`305
`
`307
`
`309
`
`311
`
`Page 00005
`
`Page 00005
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 4 of 14
`
`US 7,747,765 B2
`
`
`
`
`_ovieheade
`
`
`
`
`(video))track
`
`
`
`
`
`RTP hint track—
`
`"M
`
`
`
`
`Ihm:_—m_-O3
`
`
`
`
`
`
`
`
`--_V--
`
`
`
`PPackeihI
`amefi
`Sfll’flpfi
`
`heals!-'ni_pnter
`
`'405
`
`PPackethi
`samp8
`'hade_IIin_teI
`
`
`
`
`TPPackeihI
`
`healer“_uintér
`
`
`
`
`
`
`FIG. 4
`
`Page 00006
`
`Page 00006
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 5 of 14
`
`US 7,747,765 B2
`
` Simple Movie—Video only
`
`£02.
`
`
`
`
`
`
`
`
`
`
`Hint Movi ‘-
`--—__-
`_TPPackeihintl
`TPPackeihi_ni|
`rFIi-TPPackeihiniI
`hem—WI
`
`headeli_lnterl
`[Minn—Le!
`sample
`lsampie
`
`
`fi_-_—_-
`
`RTP hint
`track __-
`I——-
`
`
`
`__-_-_
`-—_—__-_
`
`
`
`
`
`
`
`
`
`
`
`
`5%
`
`Page 00007
`
`Page 00007
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 6 of 14
`
`US 7,747,765 B2
`
`602
`
` CUENT
`
`COMPUTER
`
`SYSTEM
`
`
`
`
` INTERNET
`
`CUENT
`COMPUTER
`
`
`
`SYSTEM
`
`\NEB
`
`SERVER
`
`SYSTEM
`
`
`
`
`
`NERNORK
`
`INTERFACE
`
`CUENT
`
`
`
`
`CUENT
`COMPUTER
`COMPUTER
`
`
`
`
`
`
`
`618
`620
`
`SYSTEM
`
`SYSTEM
`
`FIG. 6
`
`Page 00008
`
`Page 00008
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 7 of 14
`
`US 7,747,765 B2
`
`652
`
`PROCESSOR
`
`MEMORY
`
`554
`
`656
`
`658
`
`DISPLAY
`CONTROLLER
`
`MASS
`MEMORY
`
`I/O
`CONTROLLER
`
`664
`
`662
`
`DISPLAY
`
`660
`
`”0
`
`
`
`
`
`
`
`
`
`
`
`DIGITAL PROCESSING
`SYSTEM
`
`
`@
`
`MODEM
`
`OR
`
`NETWORK
`
`668
`
`INTERFACE
`
`FIG. 7
`
`Page 00009
`
`Page 00009
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 8 of 14
`
`US 7,747,765 B2
`
`
`
`MEDIA
`
`
`PROCESSING
`
`UNIT
`
`
`
`
`
`
`
`
`
`684
`
`CLIENT DATA
`DATA
`
`PROCESSING SYSTEM
`
`682
`COMMUN'CAT'ON
`
`— 7
`LINK 686
`
`
`SYSTEM 680 /
`
`FIG. 8
`
`688
`
`HINT GENERATION
`
`AND PROCESSING UNIT
`
`690
`
`MEDIA PROCESSING
`
`UNIT
`
`692
`DATA COMMUNICATION
`UNIT
`
`SERVER fig
`
`
`
`
`
`
`
`Page 00010
`
`Page 00010
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 9 of 14
`
`US 7,747,765 B2
`
`MEDIA
`
`PROCESSING
`UNIT
`
`684
`
`HINT PROCESSING
`UNIT
`
`702
`
`704
`
`MEDIA PROCESSING
`
`UNIT
`
`DATA
`CLIENT DATA
`PROCESSING SYSTEM
`COMMUN'CAT'ON
`682
`
`—
`LINK 686
`
`
`706
`DATA COMMUNICATION
`
`
`
`
`
`
`
`UNIT
`
`
`
`
`SERVER m
`
`SYSTEM 696 /
`
`DATA
`
`COMMUNICATION
`LINK 708
`
`
`
`712
`
`HINT GENERATION
`UNIT
`
`
`
`
`GENERATOR 119
`
`
`
`FIG. 9
`
`Page 00011
`
`Page 00011
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 10 of 14
`
`US 7,747,765 B2
`
`DETERMINE MEDIA FORMAT (IF MORE THAN
`ONE FORMAT IS/WILL BE USED)
`
`720
`
`DETERMINE DESIRED DATA COMMUNICATION
`PROTOCOL(S) (IF MORE THAN ONE PROTOCOL
`IS/WILL BE USED)
`
`722
`
`724
`
`BASED ON THE MEDIA FORMAT AND DESIRED
`DATA COMMUNICATION PROTOCOL(S), CREATE
`AND STORE HINTS FOR PACKETIZING A
`
`TIME RELATED SEQUENCE OF MEDIA DATA
`
`OPTIONALLY TRANSMIT HINTS TO ANOTHER
`DIGITAL PROCESSING SYSTEM
`
`726
`
`FIG. 10
`
`Page 00012
`
`Page 00012
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 11 of 14
`
`US 7,747,765 B2
`
`RECEIVE, AT A RECEIVING SYSTEM,
`MEDIA DATA TRANSMITTED TO THE
`
`WITH "HINTS"
`
`RECEIVING SYSTEM IN ACCORDANCE
`
`730
`
`PRESENT, AT THE RECEIVING SYSTEM,
`A MEDIA OBJECT REPRESENTED BY
`
`THE MEDIA DATA
`
`732
`
`OPTIONALLY STORE THE MEDIA DATA
`AS A MEDIA FILE
`
`734
`
`OPTIONALLY REASSEMBLE THE STORED
`MEDIA FILE
`
`735
`
`FIG. 11
`
`Page 00013
`
`Page 00013
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 12 of 14
`
`US 7,747,765 B2
`
`GENERATOR
`OS
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`HINT CREATION
`ROUTINE(S)
`
`746
`
`CREATED HINTS
`
`INFORMATION RELATING
`TO ONE OR MORE DATA
`COMMUNICATION PROTOCOLS
`AND/OR MEDIA FORMATS
`
`MACHINE-READABLE
`
`STORAGE MEDIUM
`740
`
`FIG. 12
`
`
`fl
`
`SERVER OS
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`MEDIA DATA
`
`MEDIA PACKETIZATION
`ROUTINE(S), BASED
`ON HINTS
`
`MACHINE-READABLE
`
`STORAGE MEDIUM
`
`FIG. 13
`
`Page 00014
`
`Page 00014
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 13 of 14
`
`US 7,747,765 B2
`
`
`
`
`RECEIVING SYSTEM
`OS
`
`(OPTIONAL)
`MEDIA DATA
`
`774
`
`776
`
`778
`
`MEDIA PRESENTATION
`
`782
`
`
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`
`
`
`
`
`
`ROUTINE(S)
` 772
`
`STORAGE MEDIUM
`$39103;sz
`FIE-ASSEMBLY ROUTINE
`18.9
`
`
`
`
`MACHINE-READABLE
`
`FIG. 14
`
`Page 00015
`
`Page 00015
`
`

`

`US. Patent
`
`Jun. 29, 2010
`
`Sheet 14 of 14
`
`US 7,747,765 B2
`
`HINT PACKET
`
`MEDIA DATA PACKET
`
`fl
`
`DATA STORAGE AND/OR
`
`COMMUNICATION MEDIUM
`
`FIG. 15
`
`Page 00016
`
`Page 00016
`
`

`

`US 7,747,765 B2
`
`1
`METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`This application is a continuation application of US.
`patent application Ser. No. 10/789,582, now issued as US.
`Pat. No. 7,366,788, filed Feb. 26, 2004, which is a continua-
`tion of US. patent application Ser. No. 10/177,119, filed Jun.
`21, 2002, now issued as US. Pat. No. 6,714,984, which is a
`continuation of US. patent application Ser. No. 09/139,378,
`filed Aug. 25, 1998, now issued as US. Pat. No. 6,453,355,
`which claims the benefit of US. Provisional Application No.
`60/071,566, filed Jan. 15, 1998.
`
`FIELD OF THE INVENTION
`
`The present invention relates to methods and apparatuses
`for preparing time related sequences of media data for trans-
`mission, and more particularly to packetized transmission of
`such media data
`
`INTRODUCTION AND BACKGROUND
`
`There are various different file structures used today to
`store time-based media: audio formats such as AIFF, video
`formats such as AVI, and streaming formats such as RealMe-
`dia. One reason that such file structures are different is their
`
`different focus and applicability. Some of these formats are
`sufficiently relatively widely accepted, broad in their appli-
`cation, and somewhat simple to implement, and thus, may be
`used not only for content delivery but also as interchange
`formats. Foremost among these general formats is the Quick-
`Time file format. It is used today in the majority of web sites
`serving time-based data; in the majority of authoring envi-
`ronments, including professional ones; and on the majority of
`multimedia CDROM titles.
`
`The QuickTime media layer supports the efficient display
`and management of general multimedia data, with an empha-
`sis on time-based material (video, audio, etc.). The media
`layer uses the QuickTime file format as the storage and inter-
`change format for media information. The architectural capa-
`bilities of the layer are generally broader than the existing
`implementations, and the file format is capable of represent-
`ing more information than is currently demanded by the exist-
`ing QuickTime implementations.
`In contrast to formats such as AVI, which were generally
`designed to support local random access of synchronized
`media, QuickTime allows systems to manage the data, rela-
`tionships and timing of a general multimedia presentation. In
`particular, the QuickTime file format has structures to repre-
`sent the temporal behavior of general time-based streams, a
`concept which covers the time-based emission of network
`packets, as well as the time-based local presentation of mul-
`timedia data.
`
`The existing QuickTime file format is publicly described
`by Apple Computer in the May 1996 File format specifica-
`tion, which may be found at the QuickTime site, <http://
`.www.apple.com/quicktime>.
`One aspect ofthe QuickTime file format is the concept that
`the physical structure of media data (the layout in disk
`records) is independent of, and described by, a logical struc-
`ture for the file. The file is fully described by a set of “movie”
`meta-data. This meta-data provides declarative, structural
`and temporal information about the actual media data.
`The media data may be in the same file as the description
`data, (the “movie” meta-data), or in other file(s). A movie
`structured into one file is commonly called “flat”, and is
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`self-contained. Non-flat movies canbe structured to reference
`some, or all, of the media data in other files.
`As such, the format is generally suited for optimization in
`different applications. For example, when editing (compos-
`iting), data need not be rewritten as edits are applied and
`media is re-ordered; the meta-data file may be extended and
`temporal mapping information adjusted. When edits are com-
`plete, the relevant media data and meta-data may be rewritten
`into a single, interleaved, and optimized file for local or
`network access. Both the structured and the optimized files
`are valid QuickTime files, and both may be inspected, played,
`and reworked.
`
`The use of structured (“non-flat”) files enables the same
`basic media data to be used and re-used in any number of
`presentations. This same advantage applies when serving, as
`will be seen below.
`
`In both editing and serving, this also permits a number of
`other files to be treated as part of a movie without copying the
`media data. Thus editing and serving may be done directly
`from files such as Sun Microsystem’s “au” audio format or
`the AVI video format, greatly extending the utility of these
`formats.
`
`The QuickTime file is divided into a set of objects, called
`atoms. Each object starts with an atom header, which declares
`its size and type:
`
`class Atom {
`int(3 2)
`size;
`char
`typew;
`byte
`contents[ ];
`
`}
`
`The size is in bytes, including the size and type header
`fields. The type field is four characters (usually printable), to
`permit easy documentation and identification. The data in an
`object after the type field may be fields, a sequence of con-
`tained objects, or both.
`A file therefore is simply a sequence of objects:
`
`class File {
`Atom[ ];
`
`}
`
`The two important top-level objects are the media-data
`(mdat) and the meta-data (moov).
`The media-data object(s) contain the actual media (for
`example, sequences of sound samples). Their format is not
`constrained by the file format; they are not usually objects.
`Their format is described in the meta-data, not by any decla-
`rations physically contiguous with them. So, for example, in
`a movie consisting solely of motion-JPEG, JPEG frames are
`stored contiguously in the media data with no intervening
`extra headers. The media data within the media data objects is
`logically divided into chunks; however, there are no explicit
`chunk markers within the media data.
`
`When the QuickTime file references media data in other
`files, it is not required that these ‘secondary’ files be formatted
`according to the QuickTime specification, since such media
`data files may be formatted as if they were the contents of a
`media object. Since the QuickTime format does not neces-
`sarily require any headers or other information physically
`contiguous with the media data, it is possible for the media
`data to be files which contain ‘foreign’ headers (e.g. UNIX
`“.au” files, or AVI files) and for the QuickTime meta-data to
`
`Page 00017
`
`Page 00017
`
`

`

`US 7,747,765 B2
`
`3
`contain the appropriate declarative information and reference
`the media data in the ‘foreign’ file. In this way the QuickTime
`file format can be used to update, without copying, existing
`bodies of material in disparate formats. The QuickTime file
`format is both an established format and is able to work with,
`include, and thereby bring forward, other established formats.
`Free space (e.g. deleted by an editing operation) can also be
`described by an object. Software reading a file that includes
`free space objects should ignore such free space objects, as
`well as objects at any level which it does not understand. This
`permits extension of the file at Virtually any level by intro-
`ducing new objects.
`The primary meta-data is the movie object A QuickTime
`file has exactly one movie object which is typically at the
`beginning or end of the file, to permit its easy location:
`
`class Movie {
`int(3 2)
`char
`MovieHeader
`contents
`
`}
`
`size;
`type[4] = ‘moov’;
`mh;
`Atom[ ];
`
`The movie header provides basic information about the
`overall presentation (its creation date, overall timescale, and
`so on). In the sequence of contained objects there is typically
`at least one track, which describes temporally presented data.
`
`class Track {
`int(32)
`char
`TrackHeader
`contents
`
`}
`
`size;
`type[4] = ‘trak’;
`th;
`Atom[ ];
`
`The track header provides relatively basic information
`about the track (its ID, timescale, and so on). Objects con-
`tained in the track might be references to other tracks (e. g. for
`complex compositing), or edit lists. In this sequence of con-
`tained objects there may be a media object, which describes
`the media which is presented when the track is played.
`The media object contains declarations relating to the pre-
`sentation required by the track (e.g. that it is sampled audio, or
`MDI, or orientation information for a 3D scene). The type of
`track is declared by its handler:
`
`class handler {
`int(32)
`size;
`char
`type[4] = ‘hdlr’;
`int(8)
`version;
`bit(24)
`flags;
`char
`handlertype[4]; -- mhlr for media handlers
`char
`handlersubtype[4] -- vide for video, soun for
`audio
`char
`bit(32)
`bit(32)
`string
`
`manufacturer[4] ;
`handlerfiags;
`handlerflagsmask;
`componentname;
`
`}
`
`Within the media information there is likewise a handler
`
`declaration for the data handler (which fetches media data),
`and a data information declaration, which defines which files
`contain the media data for the associated track. By using this
`declaration, movies may be built which span several files.
`
`4
`
`At the lowest level, a sample table is used which relates the
`temporal aspect of the track to the data stored in the file:
`
`class sampletable {
`size;
`int(32)
`type[4] = ‘stbl’;
`char
`sampledescription sd;
`timetosample
`tts;
`syncsampletable
`syncs;
`sampletochunk
`stoc;
`samplesize
`ssize;
`chunkoffset
`coffset;
`shadowsync
`ssync;
`
`}
`
`The sample description contains information about the
`media (e.g. the compression formats used in video). The
`time-to-sample table relates time in the track, to the sample
`(by index) which should be displayed at that time. The sync
`sample table declares which of these are sync (key) samples,
`not dependent on other samples.
`The sample-to-chunk object declares how to find the media
`data for a given sample, and its description given its index:
`
`class sampletochunk {
`int(32)
`size;
`char
`type[4] = ‘stsc’;
`int(8)
`version;
`bits(24)
`flags;
`int(32)
`enttycount;
`for (int i=0; i<enttycount; i++) {
`int(32)
`firstchunk;
`int(32)
`samplesperchunk;
`int(32)
`sampledescriptionindex;
`
`}
`
`}
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`The sample size table indicates the size of each sample. The
`chunkoffset table indicates the offset into the containing file
`of the start of each chunk.
`
`40
`
`Walking the above-described structure to find the appro-
`priate data to display for a given time is fairly straightforward,
`generally involving indexing and adding. Using the sync
`table, it is also possible to back-up to the preceding sync
`sample, and roll forward ‘silently’ accumulating deltas to a
`desired starting point.
`FIG. 1 shows the structure of a simple movie with one
`track. A similar diagram may be found in the QuickTime file
`format documentation, along with a detailed description of
`the fields of the various objects. QuickTime atoms (objects)
`are shown here with their type in a gray box, and a descriptive
`name above. This movie contains a single video track. The
`frames of video are in the same file, in a single chunk of data.
`It should be noted that the ‘chunk’ is a logical construct only;
`it is not an object. Inside the chunk are frames of video,
`typically stored in their native form. There are no required
`headers or fields in the video frames themselves.
`
`FIG. 2 is a diagram of a self-contained file with both an
`audio and a video track. Fewer of the atoms are shown here,
`for brevity; the pointers from the tracks into the media data
`are, of course, the usual sample table declarations, which
`include timing information.
`The QuickTime file format has a number of advantages,
`including:
`1) Scalability for size and bit-rates. The meta data is flex-
`ible, yet compact. This makes it suitable for small down-
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Page 00018
`
`Page 00018
`
`

`

`US 7,747,765 B2
`
`5
`loaded movies (e.g. on the Internet) as well as providing
`the basis for a number of high-end editing systems.
`2) Physical structure is independent of the logical and
`temporal structure. This makes it possible to optimize
`the physical structure differently depending on the use
`the file will have. In particular, it means that a single file
`format is suitable for authoring and editing; download-
`ing or placing on CDROMs; and for streaming.
`3) The file format has proven capable of handling a very
`broad variety of codec types and track types, including
`many not known at the time the format was designed.
`This proven ability to evolve in an upwards-compatible
`fashion is fundamental to the success of a storage for-
`mat.
`
`Scalable, or layered, codecs can be handled in a number of
`ways in the QuickTime file format. For a streaming protocol
`which supports scalability, the samples may be tagged with
`the layer or bandwidth threshold to be met for transmitting the
`samples.
`Tracks which form a set of alternatives (e.g. different natu-
`ral language sound tracks) can be tagged so that only one is
`selected for playback. The same structure can be used to
`select alternatives for streaming (e.g. for language selection).
`This capability is described in further detail in the QuickTime
`file format.
`
`When QuickTime displays a movie or track, the appropri-
`ate media handler accesses the media data for a particular
`time. The media handler must correctly interpret the data
`stream to retrieve the requested data. For example, with
`respect to video media, the media handler typically traverses
`several atoms to find the location and size of a sample for a
`given media time. The media handler may perform the fol-
`lowing:
`1. Determine the time in the media time coordinate system.
`2. Examine the time-to-sample atom to determine the
`sample number that contains the data for the specified
`time.
`
`3. Scan the sample-to-chunk atom to discover which chunk
`contains the sample in question.
`4. Extract the offset to the chunk from the chunk offset
`atom.
`
`5. Find the offset within the chunk and the sample’ s size by
`using the sample size atom.
`It is often desirable to transmit a QuickTime file or other
`types of time related sequences of media data over a data
`communication medium, which may be associated with a
`computer network (e.g. the Internet). In many computer net-
`works, the data which is transmitted into the network should
`generally be in a packet
`form. Normally,
`time related
`sequences of media data are not in the proper packetized
`format for transmission over a network. For example, media
`data files in the QuickTime format are not in a packetized
`format. Thus, there exists a need to collect the data, some-
`times referred to as streaming data, into packets for transmis-
`sion over a network.
`
`One prior approach to address the problem of transmitting
`time related sequences ofmedia data over a network is to send
`the media file over the network using a network or transmis-
`sion protocol, such as the Hypertext Transfer Protocol
`(HTTP). Thus, the media file itself is sent from one computer
`system over the network to another computer system. How-
`ever, there may be no desire to retain the media file at the
`receiving computing system. That is, when the media file is
`received and viewed or listened to at the receiving computer
`system, there may be no desire by the user of that receiving
`computer system to store a copy ofthe file, for example, if the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`receiving computing system is a network computer or a com-
`puter with low storage capacity.
`Another alternative approach to solving the problem of
`how to collect data for transmission by packets over a network
`is to prepare a file which contains the network protocol data
`units in the file for a particular transmission protocol. In a
`sense, such a file may be considered a packetized file which is
`stored in essentially the same format as it will be transmitted
`according to the particular transmission protocol. Performing
`this operation generally involves storing the file in a pack-
`etized form for a particular network protocol at a particular
`data transmission rate and a particular media file format.
`Thus, for each different transmission protocol at a particular
`data transmission rate, the file will essentially be replicated in
`its packetized form The fixed form of such files may restrict
`their applicability/compatibility and make it difficult to view
`such files locally. Thus, such an approach may greatly
`increase storage requirements in attempting to provide the file
`in various transmission protocols at various different data
`transmission rates. Moreover, each packetized file generated
`according to this alternative prior approach is generally lim-
`ited to a particular media file format, and thus, other media file
`formats for the same media object (e.g. a digital movie) are
`typically packetized and stored on the sending computer sys-
`tem.
`
`Yet another approach to solving the problem of how to
`stream time related sequences of media data is to perform the
`packetization of the media data when required on the trans-
`mitting system according to the particular transmission pro-
`tocol which is desired. This processing requires, in many
`cases, a relatively considerable amount oftime, and thus, may
`slow the performance of the transmitting system.
`Thus, it is desirable to provide an improved method and
`apparatus for transmitting time related sequences of media
`data
`
`SUMMARY OF THE INVENTION
`
`The present invention provides methods and apparatuses
`for processing media data transmitted in a data communica-
`tion medium. In one embodiment, a digital processing system
`receives a time related sequence ofmedia data provided to the
`digital processing system based on a set of data, wherein the
`set of data indicates a method to transmit the time related
`
`sequence of media data according to a transmission protocol.
`The set of data, itself, is a time related sequence of data
`associated with the time related sequence of media data. The
`time related sequence of media data may be presented and/or
`stored by the digital processing system.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 shows an example ofthe structure of a simple movie
`with one track in the prior art.
`FIG. 2 is an example of a self-contained movie file of the
`prior art.
`FIG. 3 is a flowchart showing one example of a method
`according to the present invention.
`FIG. 4 shows an example of a hint track of the present
`invention.
`
`FIG. 5 shows another example of a hint track ofthe present
`invention.
`
`FIG. 6 is a diagram of a network of computer systems in
`which media data may be exchanged and/or processed,
`according to one embodiment of the present invention.
`
`Page 00019
`
`Page 00019
`
`

`

`US 7,747,765 B2
`
`7
`FIG. 7 is a block diagram of a digital processing system
`which may be used in accordance with one embodiment ofthe
`present invention.
`FIG. 8 is a block diagram of a system that utilizes hints to
`transfer media data, according to one embodiment of the
`invention.
`
`FIG. 9 is a block diagram of a system that utilizes hints to
`transfer media data, according to one embodiment of the
`invention.
`
`FIG. 10 is a flow diagram illustrating a method for gener-
`ating hints for providing media data transmission, according
`to one embodiment of the invention.
`
`FIG. 11 is a flow diagram illustrating a method of process-
`ing media data received by a receiving system in accordance
`with hints, according to one embodiment of the invention.
`FIG. 12 is an example of a machine readable storage
`medium that may be accessed by a digital processing system,
`such as a generator, according to one embodiment of the
`invention.
`
`FIG. 13 is an example of a machine readable storage
`medium that may be accessed by a digital processing system,
`such as a server, according to one embodiment of the inven-
`tion.
`
`FIG. 14 is an example of a machine readable storage
`medium that may be accessed by a digital processing system,
`such as a receiving system or other digital processing system,
`according to one embodiment of the invention.
`FIG. 15 is a diagram of a data storage and/or communica-
`tion medium having stored/transported thereon media and
`hint information, according to one embodiment of the inven-
`tion.
`
`DETAILED DESCRIPTION
`
`The present invention provides methods and apparatuses
`for allowing the transmission, and particularly the packetized
`transmission of time related sequences of media data, which
`may include, for example, video, audio, video and audio, etc.,
`over a communication media, such as in a computer network.
`In one embodiment of the present invention, a digital pro-
`cessing system creates a set of data for indicating how to
`transmit a time related sequence of media data according to a
`transmission protocol. Typically, this set of data is stored on a
`storage device coupled to the digital