`Jones et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,028,080 B2
`*Sep. 27, 2011
`
`US008028080B2
`
`(54) METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`(75)
`
`Inventors: Anne Jones, Redwood City, CA (US);
`Jay Geagan, San Jose, CA (US); Kevin
`L. Gong, Sunnyvale, CA (US); Alagu
`Periyannan, San Francisco, CA (US);
`David W. Singer, San Francisco, CA
`98>
`(73) Assignee: Apple Inc., Cupertino, CA (US)
`
`(58) Field of Classification Search ................ .. 709/230,
`709/231; 370/394
`See application file for complete search history.
`
`(56)
`
`Referenees Cited
`
`U~S- PATENT DOCUMENTS
`3,873,777 A
`3/1975 Uehara et al.
`2523:5212:
`113;:
`,
`,
`4’907’224 A
`3/1990 ‘$001635 et a1‘
`(Continued)
`
`itt et a .
`
`e
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`This patent is subject to a terminal dis-
`Claim”
`
`CA
`
`FOREIGN PATENT DOCUMENTS
`1298632
`4/1992
`(Continued)
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 12/822,152
`
`(22)
`
`Filed:
`
`Jun. 23, 2010
`
`(65)
`
`Prior Publication Data
`US 2010/0262713 A1
`Oct. 14,2010
`
`Related US. Application Data
`
`(63) Colntinuation of application No. 11/497,038, filed on
`Ju . 31 2006 now Pat. No. 7 747 765 which is a
`5
`5
`5
`5
`5
`continuation of application No. 10/789,582, filed on
`Feb. 26, 2004, now pat. NO. 7,366,788’ which is a
`continuation of application No. 10/177,119, filed on
`Jun. 21, 2002, now Pat. No. 6,714,984, which is a
`continuation of application No. 09/139,378, filed on
`Aug. 25, 1998, now Pat. NO. 6,453,355.
`.
`.
`.
`.
`Pf0V1S10I1al appl1CaUOI1 N0. 60/071,566, filed 011 Jan.
`15, 1998-
`
`(60)
`
`(51)
`
`Int CL
`(2006.01)
`G06F 15/16
`(52) U.S. Cl.
`....................... .. 709/230; 709/231; 370/394
`
`Susie J. Wee et al., “Secure Scalable Streaming Enabling Transcod-
`ing without Decryption”, Proceedings 2002 International Confer-
`ence on Image Processing, ICIP, Oct. 7, 2001, Vol. 1 of3, Conf. 8, pp.
`437-440, IEEE, USA.
`
`on inue
`(C t,
`
`d)
`
`Primary Examiner — Krisna Lim
`(74) Attorney, Agent, 0rFirm — Blakely, Sokoloff, Taylor&
`
`Zafman’ LLP
`ABSTRACT.
`(
`)
`Methods and apparatuses forprocessing media data transmit-
`ted In a. data °.°mm‘%m°a‘}°n med1um' A dlgltal Processlng
`SYS““:‘(‘1‘1§ Pr°¥11d‘:1‘IW1‘l‘a“me “Rated Sequence Ofmedla data
`grow :1 tO.t E lglta prOC.eSS.mg System based on a S.et of
`.ata, w ereint e set ofdata indicates amethod to transmit the
`time related sequence of media data according to a transmis-
`(1
`sion protocol. The set of data, itself, is atime related se uence
`of data associated with the time related sequence of media
`data. The time related sequence of media data may be pre-
`sented and/or stored by the digital processing system.
`
`20 Claims, 14 Drawing Sheets
`
`/ 300
`
`DETERMINE MEDIA I=II.E FORMAT
`
`DETERMINE DESIRED TRANsMIssIoN PROTOCOL(S)
`
`CREATE AND STORE "HINTS' FOR PACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA F|LE(S) (PACKEIIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA F|LE(S) AND HINTS To A RECEIVING svsTEM; DATA Is
`TI=IANsMn-TED BY PACKEIIZING THE MEDIA DATA ACCORDING To
`THE HINTS
`
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`OPTIONALLV REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`RECEIVING SYSTEM
`
`3°‘
`
`3'33
`
`305
`
`3°’
`
`309
`
`3“
`
`Apple Exhibit 4241
`
`Apple v. SightSound Technologies
`CBM2013-00020
`
`Page 00001
`
`Apple Exhibit 4241
`Apple v. SightSound Technologies
`CBM2013-00020
`Page 00001
`
`
`
`US 8,028,080 B2
`Page 2
`
`>>>>>>D>D>>>>>D>D>>>>>>>>D>D>D>D>D>>D>D>D>D>D>>>D>D>D>D>>D>
`
`U.S. PATENT DOCUMENTS
`10/1993
`Jurkevich et al.
`6/1994
`Wasilewski et al.
`11/1994
`Siracusa
`12/1994
`Siracusa et al.
`4/1995
`Chung et al.
`9/1995
`Delpuch et al.
`3/1996
`Hulen et al.
`8/1996
`Saalfrank et al.
`11/1996
`Keckler et al.
`4/1997
`Richter et al.
`4/1997
`Zarmer et al.
`8/1997
`Goldberg et al.
`8/1997
`Porter et al.
`11/1997
`Gaytan et al.
`12/1997
`Donahue et al.
`6/1998
`Chaddha et al.
`6/1998
`Portuesi
`7/1998
`Monteiro et al.
`7/1998
`Meyer
`8/1998
`Hamilton et al.
`9/1998
`Ludwig et al.
`10/1998
`Throckmorton et al.
`10/1998
`Higashimura et al.
`11/1998
`Davis et al.
`1/1999
`Perkins et al.
`1/1999
`Porter et al.
`6/1999
`Kouloheris et al.
`7/1999
`Goetz et al.
`9/1999
`Goetz et al.
`10/1999
`Arazi et al.
`11/1999
`Portuesi
`11/1999
`Richter et al.
`Jones
`4/2000
`8/2000
`Kalmanek et al.
`8/2000
`Yoshida et al.
`8/2000
`Weaver et al.
`9/2000
`Weaver et al.
`10/2000
`Jones et al.
`10/2000
`Weaver et al.
`12/2000
`Oda et al.
`1/2001
`Schuster et al.
`1/2001
`Neumann et al.
`Barton
`12/2001
`8/2002
`Nakamura et al.
`9/2002
`Jones et al.
`1/2003
`Jones et al.
`
`5,251,209
`5,319,707
`5,365,272
`5,371,547
`5,404,469
`5,448,568
`5,497,373
`5,544,198
`5,574,939
`5,623,490
`5,625,818
`5,655,117
`5,659,539
`5,689,509
`5,694,334
`5,768,535
`5,774,666
`5,778,187
`5,784,277
`5,799,150
`5,802,294
`5,818,441
`5,826,024
`5,838,678
`5,859,660
`5,864,682
`5,915,094
`5,928,330
`5,956,729
`5,966,120
`5,987,509
`5,995,491
`6,055,246
`6,098,188
`6,104,859
`6,112,226
`6,119,154
`6,134,243
`6,138,147
`6,157,674
`6,175,871
`6,175,872
`6,327,418
`6,438,172
`6,453,355
`6,512,778
`
`6/2003 Weaver et al.
`6,578,070 B1
`3/2004 Jones et al.
`6,714,984 B2
`4/2004 Jones et al.
`6,717,952 B2
`6/2004 Jones et al.
`6,744,763 B1
`6/2004 Guedalia
`6,745,226 B1
`12/2004 Jones et al.
`6,829,648 B1
`1/2007 Wang et al.
`7,161,957 B2
`4/2008 Jones et al.
`7,366,788 B2
`3/2002 Van Der Schaar
`2002/0037037 A1
`9/2005 Han
`2005/0195899 A1
`9/2005 Han
`2005/0195900 A1
`1/2007 Singer et al.
`2007/0022215 A1
`FOREIGN PATENT DOCUMENTS
`2197323
`10/2001
`2387254
`3/2003
`0 497 449 A2
`8/1992
`0 702 309 A1
`3/1996
`1 458 196 A2
`9/2004
`9101928 A
`4/1997
`9200158 A
`7/1997
`WO 97/22201
`6/1997
`WO 97/25817 A1
`7/1997
`W0 02/054284
`7/2002
`
`CA
`CA
`EP
`EP
`EP
`JP
`JP
`W0
`W0
`W0
`
`OTHER PUBLICATIONS
`
`International Search Report, PCT/US2006/028275, Dec. 18, 2006,
`11 pages.
`Aaron E.Walsh, “Multimedia to the MACS”, Dr. Dobb’ s Journal, Jul.
`1992, pp. 76, 78-80.
`Paul England et al., “RAVE: Real-Time Services for the Web”, Com-
`puter Networks and ISDN Systems, May 1996, pp. 1547-1558.
`PCT International Search Report for PCT International Application
`No. PCT/US99/00953, mailed Jul. 26, 1999.
`PCT International Search Report for PCT International Application
`No. PCT/US99-00954 mailed Jul. 26, 1999.
`PCT International Search Report for PCT International Application
`No. PCT/US99-00955 mailed Jul. 26, 1999.
`Song, Jun., “Synchronizing Feature of Multimedia”, Today ’s Elec-
`tronics, Jan. 18, 1997, pp. 30-31 in Chinese.
`Song, Jun., “Synchronizing Feature of Multimedia”, Today ’s Elec-
`tronics, Jan. 18, 1997, translated into English (p. 1-6).
`Susie J. Wee et al., “Secure Scalable Video Streaming for Wireless
`Networks”, IEEE International Conference on Acoustics, Speech,
`and Signal Processing, Salt Lake City, Utah, May 2001, 4 pages.
`
`Page 00002
`
`Page 00002
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 1 of 14
`
`US 8,028,080 B2
`
`media data
`U
`
`DC!
`
`DE!
`
`DE!
`
`D
`
`DE
`
`DE
`
`DE
`
`U
`
`U
`
`
`
`
`
`FIG. 1
`
`Prior Art
`
`Page 00003
`
`Page 00003
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 2 of 14
`
`US 8,028,080 B2
`
`movie header
`
`(video) track
`
`(audio) track -—-
`
`Illlifi
`
`Imm-
`
`Page 00004
`
`Page 00004
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 3 of 14
`
`US 8,028,080 B2
`
`301
`
`303
`
`305
`
`307
`
`309
`
`311
`
`Page 00005
`
`/ 300
`
`DETERMINE MEDIA FILE I=oRMAT
`
`DETERMINE DESIRED TRANSMISSION PFIOTOCOL(S)
`
`CREATE AND STORE "HINTS" FOR PACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA FILE(S) (PACKETIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`MEDIA DATA)
`
`
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA FILE(S) AND HINTS TO A RECEIVING SYSTEM; DATA IS
`TRANSMITTED BY PACKETIZING THE MEDIA DATA ACCORDING TO
`THE HINTS
`
`
`
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`
`
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`RECEIVING SYSTEM
`
`FIG. 3
`
`Page 00005
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 4 of 14
`
`US 8,028,080 B2
`
`movie heade
`
`I
`
`
`
`
`
`video track
`
`trak
`
`RTP hint
`
`£3
`track88
`
`
`
`'-A
`""""""""""""'—"
`:1 HT Packethint
`-
`3
`RTPPad(ethint
`-
`u
`HT Packeihint
`
`
`
`
`
`
`
`
`
`
`sampe
`
`1;,
`
`sampe
`
`_
`
`sampe
`
`.,
`
`
`
`;
`
`.
`
`
`
`
`
`
`
`T j T
`
`405
`
`FIG. 4
`
`Page 00006
`
`Page 00006
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 5 of 14
`
`US 8,028,080 B2
`
`Simple Movie—Video only
`Movie
`
`
`
`
`
`
`
`‘-
`
`I
`
`CI
`
`I]
`
`
`
`
`
`
`
`(video) track_
`
`DM:
`D ..
`§'’““i
`
`Hinmovs Pmeia-damn —
`
`
`
`—Z—— I
`j
`d
`iiTPPackeihini
`HTPPackeihini
`RTPPackeiii{ii
`m» I I
`
`-
`
`sampe
`
`sampe
`
`
`
`—_——_
`mm
`RTP hint
`track _—-
`_—
`
`
`
`
`
`
`
`FIG. 5
`
`Page 00007
`
`m at
`
`
`
`
`
`Page 00007
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 6 of 14
`
`US 8,028,080 B2
`
`602
`
`606
`
`CUENT
`
`COMPUTER @
`
`SYSTEM
`
`604
`CLIENT
`
`608
`
`COMPUTER M
`
`SYSTEM
`
`6Io
`
`GATEWAY
`SYSTEM
`
`612
`
`614
`
`515
`
`NETWORK
`INTERFACE
`
`NETWORK
`INTERFACE
`
`CLIENT
`COMPUTER
`
`SYSTEM
`
`CLIENT
`COMPUTER
`
`SYSTEM
`
`618
`
`620
`
`FIG. 6
`
`INTERNET
`
`WEB
`
`SERVER
`SYSTEM
`
`\
`
`600
`
`Page 00003
`
`Page 00008
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 7 of 14
`
`US 8,028,080 B2
`
`PROCESSOR
`
`652
`
`
`
`656
`
`MEMORY
`
`654
`
`
`
`
`—
`
`658
`
`
`
`DISPLAY
`CONTROLLER
`
`MASS
`MEMORY
`
`I/O
`CONTROLLER
`
`664
`
`662
`
`DISPLAY
`
`660
`
`V0
`DEVICES
`
`666
`
`
`
`
`
`
`
` MODEM
` 668
`
`DIGITAL PROCESSING
`SYSTEM
`
`‘J
`
`
`
`OR
`
`NETWORK
`
`
`INTERFACE
`
`
`FIG. 7
`
`Page 00009
`
`Page 00009
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 8 of 14
`
`US 8,028,080 B2
`
`
`
`AND PROCESSING UNIT
`
`MEDIA
`
` HINT GENERATION
`
`
`
`
`
`
`
`
`PROCESSING SYSTEM
`
`g8_2_
`
`C°ML",",\‘,’,£“',‘33é‘6T'°N
`
`DATA COMMUNICATION
`
`UNIT
`
`
`
`SERVER gag
`
`SYSTEM 680 '1
`
`FIG. 8
`
`Page 00010
`
`
`
`
`
`
`
`PROCESSING
`UNIT
`
`684
`
`MEDIA PROCESSING
`UNIT
`
`CLIENT DATA DATA
`
`Page 00010
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 9 of 14
`
`US 8,028,080 B2
`
`
`
`
`
`702
`
`HINT PROCESSING
`UNIT
`
`704
`
`MEDIA PROCESSING
`UNIT
`
`706
`
`DATA COMMUNICATION
`
`UNIT
`
`
`
`684
`
`DATA
`COMMUMCA-“ON
`LINK 686
`
`
`
`CLIENT DATA
`PROCESSING SYSTEM
`63
`
`
`
`
`SERVER 113
`
`SYSTEM 696 J
`
`DATA
`
`COMMUNICATION
`LINK 708
`
`
`
`
`
`712
`
`HINT GENERATION
`UNIT
`
`GENERATOR _719_
`
`FIG. 9
`
`Page 00011
`
`
`
`
`
`
`MEDIA
`PROCESSING
`
`UNIT
`
`Page 00011
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 10 0114
`
`US 8,028,080 B2
`
`DETERMINE MEDIA FORMAT (IF MORE THAN
`ONE FORMAT IS/WILL BE USED)
`
`790
`
`DETERMINE DESIRED DATA COMMUNICATION
`PROTOCOL(S) (IF MORE THAN ONE PROTOCOL
`IS/INILL BE USED)
`
`
`
`BASED ON THE MEDIA FORMAT AND DESIRED
`DATA COMMUNICATION PROTOCOL(S), CREATE
`AND STORE HINTS FOR PACKETIZING A
`
`TIME RELATED SEQUENCE OF MEDIA DATA
`
`722
`
`724
`
`OPTIONALLY TRANSMIT HINTS TO ANOTHER
`DIGITAL PROCESSING SYSTEM
`
`725
`
`FIG. 10
`
`Page 00012
`
`Page 00012
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 11 of 14
`
`US 8,028,080 B2
`
`RECEIVE, AT A RECEIVING SYSTEM,
`MEDIA DATA TRANSMITTED TO THE
`RECEIVING SYSTEM IN ACCORDANCE
`
`WITH "HINTS"
`
`THE MEDIA DATA
`
`PRESENT, AT THE RECEIVING SYSTEM,
`A MEDIA OBJECT REPRESENTED BY
`
`73°
`
`732
`
`OPTIONALLY sToRE THE MEDIA DATA
`AS A MEDIA FILE
`
`734
`
`OPTIONALLY REASSEMBLE THE sToRED
`MEDIA FILE
`
`736
`
`FIG. 11
`
`Page 00013
`
`Page 00013
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 12 of 14
`
`US 8,028,080 B2
`
`742
`
`744
`
`746
`
`762
`
`754
`
`GENERATOR
`OS
`
`HINT CREATION
`ROUT|NE(S)
`
`CREATED HINTS
`
`SERVER 03
`
`MEDIA DATA
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`INFORMATION RELATING
`TO ONE OR MORE DATA
`COMMUNICATION PROTOCOLS
`AND/OR MEDIA FORMATS
`
`MACHINE-READABLE
`STORAGE MEDIUM
`740
`
`FIG. 12
`
`NETWORK TRANSMISSION
`ROUTlNE(S)
`
`MEDIA PACKETIZATION
`ROUTINE(S), BASED
`ON HINTS
`
`MACHINE-READABLE
`
`STORAGE MEDIUM
`1%
`
`FIG. 13
`
`Page 00014
`
`Page 00014
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 13 0114
`
`US 8,028,080 B2
`
`
`
`
`
`772
`
`RECEIVING SYSTEM
`OS
`
`(OPTIONAL)
`MEDIA DATA
`
`774
`
`776
`
`(OPHONAL)
`MEDM DATA
`RE-ASSEMBLY ROUTINE
`
`
`
`
`
`778
`
`MEDIA PRESENTATION
`ROUTINE(S)
`
`782
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`
`
`339
`
`MACHINE-READABLE
`STORAGE MEDIUM
`
`FIG. 14
`
`Page 00015
`
`Page 00015
`
`
`
`U.S. Patent
`
`Sep. 27, 2011
`
`Sheet 14 of 14
`
`US 8,028,080 B2
`
`HINT PACKET
`
`MED|A DATA PACKET
`
`DATA STORAGE AND/OR
`
`COMMUNICATION MEDIUM
`_$_Q0_
`
`FIG. 15
`
`Page 00016
`
`Page 00016
`
`
`
`US 8,028,080 B2
`
`1
`METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`This application is a continuation of U.S. patent applica-
`tion Ser. No. 11/497,038, filed on Jul. 31, 2006, now U.S. Pat.
`No. 7,747,765, which is a continuation of Ser. No. 10/789,
`582, filed Feb. 26, 2004, now U.S. Pat. No. 7,366,788, which
`is a continuation of U.S. patent application Ser. No. 10/177,
`119, filed Jun. 21, 2002, now issued as U.S. Pat. No. 6,714,
`984, which is a continuation of U.S. patent application Ser.
`No. 09/139,378, filed Aug. 25, 1998, now issued as U.S. Pat.
`No. 6,453,355, which claims the benefit of U.S. Provisional
`Application No. 60/071,566, filed Jan. 15, 1998.
`
`FIELD OF THE INVENTION
`
`The present invention relates to methods and apparatuses
`for preparing time related sequences of media data for trans-
`mission, and more particularly to packetized transmission of
`such media data.
`
`INTRODUCTION AND BACKGROUND
`
`There are various different file structures used today to
`store time-based media: audio formats such as AIFF, video
`formats such as AVI, and streaming formats such as RealMe-
`dia. One reason that such file structures are different is their
`
`different focus and applicability. Some of these formats are
`sufficiently relatively widely accepted, broad in their appli-
`cation, and somewhat simple to implement, and thus, may be
`used not only for content delivery but also as interchange
`formats. Foremost among these general formats is the Quick-
`Time file format. It is used today in the majority of web sites
`serving time-based data; in the majority of authoring envi-
`ronments, including professional ones; and on the majority of
`multimedia CDROM titles.
`
`The QuickTime media layer supports the efiicient display
`and management of general multimedia data, with an empha-
`sis on time-based material (video, audio, etc.). The media
`layer uses the QuickTime file format as the storage and inter-
`change format for media information. The architectural capa-
`bilities of the layer are generally broader than the existing
`implementations, and the file format is capable of represent-
`ing more information than is currently demanded by the exist-
`ing QuickTime implementations.
`In contrast to formats such as AVI, which were generally
`designed to support local random access of synchronized
`media, QuickTime allows systems to manage the data, rela-
`tionships and timing of a general multimedia presentation. In
`particular, the QuickTime file format has structures to repre-
`sent the temporal behavior of general time-based streams, a
`concept which covers the time-based emission of network
`packets, as well as the time-based local presentation of mul-
`timedia data.
`
`The existing QuickTime file format is publicly described
`by Apple Computer in the May 1996 File format specifica-
`tion, which may be found at the QuickTime site, <http://.
`www.apple.con1/quicktime>.
`One aspect ofthe QuickTime file format is the concept that
`the physical structure of media data (the layout in disk
`records) is independent of, and described by, a logical struc-
`ture for the file. The file is fully described by a set of “movie”
`meta-data. This meta-data provides declarative, structural
`and temporal information about the actual media data.
`The media data may be in the same file as the description
`data, (the “movie” meta-data), or in other file(s). A movie
`structured into one file is commonly called “flat”, and is
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`self-contained. Non-flat movies canbe structured to reference
`some, or all, of the media data in other files.
`As such, the format is generally suited for optimization in
`different applications. For example, when editing (compos-
`iting), data need not be rewritten as edits are applied and
`media is re-ordered; the meta-data file may be extended and
`temporal mapping information adjusted. When edits are com-
`plete, the relevant media data and meta-data may be rewritten
`into a single, interleaved, and optimized file for local or
`network access. Both the structured and the optimized files
`are valid QuickTime files, and both may be inspected, played,
`and reworked.
`
`The use of structured (“non-flat”) files enables the same
`basic media data to be used and re-used in any number of
`presentations. This same advantage applies when serving, as
`will be seen below.
`
`In both editing and serving, this also permits a number of
`other files to be treated as part of a movie without copying the
`media data. Thus editing and serving may be done directly
`from files such as Sun Microsystem’s “au” audio format or
`the AVI video format, greatly extending the utility of these
`formats.
`
`The QuickTime file is divided into a set of objects, called
`atoms. Each object starts with an atom header, which declares
`its size and type:
`
`class Atom {
`int(32)
`char
`byte
`
`}
`
`size;
`type[4];
`contents[ ];
`
`The size is in bytes, including the size and type header
`fields. The type field is four characters (usually printable), to
`permit easy documentation and identification. The data in an
`object after the type field may be fields, a sequence of con-
`tained objects, or both.
`A file therefore is simply a sequence of objects:
`
`class File {
`Atom[ ];
`
`}
`
`The two important top-level objects are the media-data
`(mdat) and the meta-data (moov).
`The media-data object(s) contain the actual media (for
`example, sequences of sound samples). Their format is not
`constrained by the file format; they are not usually objects.
`Their format is described in the meta-data, not by any decla-
`rations physically contiguous with them. So, for example, in
`a movie consisting solely of motion-JPEG, JPEG frames are
`stored contiguously in the media data with no intervening
`extra headers. The media data within the media data objects is
`logically divided into chunks; however, there are no explicit
`chunk markers within the media data.
`
`When the QuickTime file references media data in other
`files, it is not required that these ‘secondary’ files be formatted
`according to the QuickTime specification, since such media
`data files may be formatted as if they were the contents of a
`media object. Since the QuickTime format does not neces-
`sarily require any headers or other information physically
`contiguous with the media data, it is possible for the media
`data to be files which contain ‘foreign’ headers (e.g. UNIX
`“.au” files, or AVI files) and for the QuickTime meta-data to
`
`Page 00017
`
`Page 00017
`
`
`
`US 8,028,080 B2
`
`3
`contain the appropriate declarative information and reference
`the media data in the ‘foreign’ file. In this way the QuickTime
`file format can be used to update, without copying, existing
`bodies of material in disparate formats. The QuickTime file
`format is both an established format and is able to work with,
`include, and thereby bring forward, other established formats.
`Free space (e.g. deleted by an editing operation) can also be
`described by an object. Software reading a file that includes
`free space objects should ignore such free space objects, as
`well as objects at any level which it does not understand. This
`permits extension of the file at virtually any level by intro-
`ducing new objects.
`The primary meta-data is the movie object. A QuickTime
`file has exactly one movie object which is typically at the
`beginning or end of the file, to permit its easy location:
`
`class Movie {
`size;
`int(32)
`type[4] = ‘moov’;
`char
`MovieHeader mh;
`contents
`Atom[ ];
`
`}
`
`The movie header provides basic information about the
`overall presentation (its creation date, overall timescale, and
`so on). In the sequence of contained objects there is typically
`at least one track, which describes temporally presented data.
`
`class Track {
`int(32)
`char
`TrackHeader
`contents
`
`}
`
`size;
`type[4] = ‘trak’;
`th;
`Atom[ ];
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`4
`
`At the lowest level, a sample table is used which relates the
`temporal aspect of the track to the data stored in the file:
`
`class sampletable {
`int(32)
`size;
`char
`type[4] = ‘stbl’;
`sampledescription
`sd;
`timetosample
`tts;
`syncsampletable
`syncs;
`sampletochunk
`stoc;
`samplesize
`ssize;
`chunkoffset
`coffset;
`shadowsync
`ssync;
`
`The sample description contains information about the
`media (e.g. the compression formats used in video). The
`time-to-sample table relates time in the track, to the sample
`(by index) which should be displayed at that time. The sync
`sample table declares which of these are sync (key) samples,
`not dependent on other samples.
`The sample-to-chunk object declares how to find the media
`data for a given sample, and its description given its index:
`
`class sampletochunk {
`int(32)
`size;
`char
`type[4] = ‘stsc’;
`int(8)
`version;
`bits(24)
`flags;
`int(3 2)
`enttycount;
`for (int i=0; i<enttycount; i++) {
`int(32)
`firstchunk;
`int(32)
`samplesperchunk;
`int(32)
`sampledescriptionindex;
`
`The track header provides relatively basic information
`about the track (its ID, timescale, and so on). Objects con-
`tained in the track might be references to other tracks (e. g. for
`complex compositing), or edit lists. In this sequence of con-
`tained objects there may be a media object, which describes
`the media which is presented when the track is played.
`The media object contains declarations relating to the pre-
`sentation required by the track (e.g. that it is sampled audio, or
`MIDI, or orientation information for a 3Dscene). The type of
`track is declared by its handler:
`
`class handler {
`int(32)
`char
`int(8)
`bit(24)
`char
`char
`char
`bit(32)
`bit(32)
`string
`
`}
`
`size;
`type[4] = ‘hdlr’;
`version;
`flags;
`-- mhlr for media handlers
`handlertype[4];
`handlersubtype[4] -- vide for video, soun for audio
`manufacturer[4] ;
`handlerflags;
`handlerflagsmask;
`componentname;
`
`Within the media information there is likewise a handler
`
`declaration for the data handler (which fetches media data),
`and a data information declaration, which defines which files
`contain the media data for the associated track. By using this
`declaration, movies may be built which span several files.
`
`The sample size table indicates the size of each sample. The
`chunkoffset table indicates the offset into the containing file
`of the start of each chunk.
`
`40
`
`Walking the above-described structure to find the appro-
`priate data to display for a given time is fairly straightforward,
`generally involving indexing and adding. Using the sync
`table, it is also possible to back-up to the preceding sync
`sample, and roll forward ‘silently’ accumulating deltas to a
`desired starting point.
`FIG. 1 shows the structure of a simple movie with one
`track. A similar diagram may be found in the QuickTime file
`format documentation, along with a detailed description of
`the fields of the various objects. QuickTime atoms (objects)
`are shown here with their type in a grey box, and a descriptive
`name above. This movie contains a single video track. The
`frames of video are in the same file, in a single chunk of data.
`It should be noted that the ‘chunk’ is a logical construct only;
`it is not an object. Inside the chunk are frames of video,
`typically stored in their native form. There are no required
`headers or fields in the video frames themselves.
`
`FIG. 2 is a diagram of a self-contained file with both an
`audio and a video track. Fewer of the atoms are shown here,
`for brevity; the pointers from the tracks into the media data
`are, of course, the usual sample table declarations, which
`include timing information.
`The QuickTime file format has a number of advantages,
`including:
`1) Scalability for size and bit-rates. The meta data is flex-
`ible, yet compact. This makes it suitable for small down-
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Page 00013
`
`Page 00018
`
`
`
`US 8,028,080 B2
`
`5
`loaded movies (e.g. on the Internet) as well as providing
`the basis for a number of high-end editing systems.
`2) Physical structure is independent of the logical and
`temporal structure. This makes it possible to optimize
`the physical structure differently depending on the use
`the file will have. In particular, it means that a single file
`format is suitable for authoring and editing; download-
`ing or placing on CDROMs; and for streaming.
`3) The file format has proven capable of handling a very
`broad variety of codec types and track types, including
`many not known at the time the format was designed.
`This proven ability to evolve in an upwards-compatible
`fashion is fundamental to the success of a storage for-
`mat.
`
`Scalable, or layered, codecs can be handled in a number of
`ways in the QuickTime file format. For a streaming protocol
`which supports scalability, the samples may be tagged with
`the layer or bandwidth threshold to be met for transmitting the
`samples.
`Tracks which form a set of alternatives (e.g. different natu-
`ral language sound tracks) can be tagged so that only one is
`selected for playback. The same structure can be used to
`select alternatives for streaming (e.g. for language selection).
`This capability is described in further detail in the QuickTime
`file format.
`
`When QuickTime displays a movie or track, the appropri-
`ate media handler accesses the media data for a particular
`time. The media handler must correctly interpret the data
`stream to retrieve the requested data. For example, with
`respect to video media, the media handler typically traverses
`several atoms to find the location and size of a sample for a
`given media time. The media handler may perform the fol-
`lowing:
`1. Determine the time in the media time coordinate system.
`2. Examine the time-to-sample atom to determine the
`sample number that contains the data for the specified
`time.
`
`3. Scan the sample-to-chunk atom to discover which chunk
`contains the sample in question.
`4. Extract the offset to the chunk from the chunk offset
`atom.
`
`5. Find the offset within the chunk and the sample’ s size by
`using the sample size atom.
`It is often desirable to transmit a QuickTime file or other
`types of time related sequences of media data over a data
`communication medium, which may be associated with a
`computer network (e.g. the Internet). In many computer net-
`works, the data which is transmitted into the network should
`generally be in a packet
`form. Normally,
`time related
`sequences of media data are not in the proper packetized
`format for transmission over a network. For example, media
`data files in the QuickTime format are not in a packetized
`format. Thus, there exists a need to collect the data, some-
`times referred to as streaming data, into packets for transmis-
`sion over a network.
`
`One prior approach to address the problem of transmitting
`time related sequences ofmedia data over a network is to send
`the media file over the network using a network or transmis-
`sion protocol, such as the Hypertext Transfer Protocol
`(HTTP). Thus, the media file itself is sent from one computer
`system over the network to another computer system. How-
`ever, there may be no desire to retain the media file at the
`receiving computing system. That is, when the media file is
`received and viewed or listened to at the receiving computer
`system, there may be no desire by the user of that receiving
`computer system to store a copy ofthe file, for example, if the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`receiving computing system is a network computer or a com-
`puter with low storage capacity.
`Another alternative approach to solving the problem of
`how to collect data for transmission by packets over a network
`is to prepare a file which contains the network protocol data
`units in the file for a particular transmission protocol. In a
`sense, such a file may be considered a packetized file which is
`stored in essentially the same format as it will be transmitted
`according to the particular transmission protocol. Performing
`this operation generally involves storing the file in a pack-
`etized form for a particular network protocol at a particular
`data transmission rate and a particular media file format.
`Thus, for each different transmission protocol at a particular
`data transmission rate, the file will essentially be replicated in
`its packetized form. The fixed form of such files may restrict
`their applicability/compatibility and make it difficult to view
`such files locally. Thus, such an approach may greatly
`increase storage requirements in attempting to provide the file
`in various transmission protocols at various different data
`transmission rates. Moreover, each packetized file generated
`according to this alternative prior approach is generally lim-
`ited to a particular media file format, and thus, other media file
`formats for the same media object (e.g. a digital movie) are
`typically packetized and stored on the sending computer sys-
`tem.
`
`Yet another approach to solving the problem of how to
`stream time related sequences of media data is to perform the
`packetization of the media data when required on the trans-
`mitting system according to the particular transmission pro-
`tocol which is desired. This processing requires, in many
`cases, a relatively considerable amount oftime, and thus, may
`slow the performance of the transmitting system.
`Thus, it is desirable to provide an improved method and
`apparatus for transmitting time related sequences of media
`data.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides methods and apparatuses
`for processing media data transmitted in a data communica-
`tion medium. In one embodiment, a digital processing system
`receives a time related sequence ofmedia data provided to the
`digital processing system based on a set of data, wherein the
`set of data indicates a method to transmit the time related
`
`sequence of media data according to a transmission protocol.
`The set of data, itself, is a time related sequence of data
`associated with the time related sequence of media data. The
`time related sequence of media data may be presented and/or
`stored by the digital processing system.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 shows an example ofthe structure of a simple movie
`with one track in the prior art.
`FIG. 2 is an example of a self-contained movie file of the
`prior art.
`FIG. 3 is a flowchart showing one example of a method
`according to the present invention.
`FIG. 4 shows an example of a hint track of the present
`invention.
`
`FIG. 5 shows another example of a hint track ofthe present
`invention.
`
`FIG. 6 is a diagram of a network of computer systems in
`which media data may be exchanged and/or processed,
`according to one embodiment of the present invention.
`
`Page 00019
`
`Page 00019
`
`
`
`US 8,028,080 B2
`
`7
`FIG. 7 is a block diagram of a digital processing system
`which may be used in accordance with one embodiment ofthe
`present invention.
`FIG. 8 is a block diagram of a system that utilizes hints to
`transfer media data, according to one embodiment of the
`invention.
`
`FIG. 9 is a block diagram of a system that utilizes hints to
`transfer media data, according to one embodiment of the
`invention.
`
`FIG. 10 is a flow diagram illustrating a method for gener-
`ating hints for providing media data transmission, according
`to one embodiment of the invention.
`
`FIG. 11 is a flow diagram illustrating a method of process-
`ing media data received by a receiving system in accordance
`with hints, according to one embodiment of the invention.
`FIG. 12 is an example of a machine readable storage
`medium that may be accessed by a digital processing system,
`such as a generator, according to one embodiment of the
`invention.
`
`FIG. 13 is an example of a machine readable storage
`medium that may be accessed by a digital processing system,
`such as a server, according to one embodiment of the inven-
`tion.
`
`FIG. 14 is an example of a machine readable storage
`medium that may be accessed by a digital processing system,
`such as a receiving system or other digital processing system,
`according to one embodiment of the invention.
`FIG. 15 is a diagram of a data storage and/or communica-
`tion medium having stored/transported thereon media and
`hint information, according to one embodiment of the inven-
`tion.
`
`DETAILED DESCRIPTION
`
`The present invention provides methods and apparatuses
`for allowing the transmission, and particularly the packetized
`transmission of time related sequences of media data, which
`may include, for example, video, audio, video and audio, etc.,
`over a communication media, such as in a computer network.
`In one embodiment of the present invention, a digital pro-
`cessing system creates a set of data for indicating how to
`transmit a time related sequence of media data according to a
`transmission protocol. Typically, this set of data is stored on a
`storage device coupled to the digital processing system. Fur-
`ther, this set of data is a time related sequence of data