(12) United States Patent
`US 6,453,355 BI
`(10) Patent No.:
`Jones et al.
`
`(45) Date of Patent: *Sep. 17, 2002
`
`USUUG453355B]
`
`(54)
`
`(75)
`
`METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`Inventors: Anne Jones, Redwood City; Jay
`Geagan, San Jose; Kevin L. Gong,
`Sunnyvale; Alagu Periyannan; David
`W. Singer, both of San Francisco, all of
`(:A (US)
`
`t 73)
`
`Assignee:
`
`Apple Computer, Ine., Cupertlno, CA
`(US)
`
`(*)
`
`Not ice:
`
`This patent issued on a continued pros-
`ecution application filed under 37 CFR
`1.53M}, and is subject to the twenty year
`patent
`term provisions of 35 U.S.C.
`1.5400(2).
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.(T. 154(b) by 0 days.
`
`(31)
`
`(33)
`
`(60)
`
`(51)
`(53)
`
`(58)
`
`(56)
`
`Appl. No.:
`Filed:
`
`097139,378
`
`Aug. 25, 1998
`
`Related U.S. Application Data
`Provisional application No. 61.JAI71,566, filed on Jan. 15,
`1998.
`
`Int. Cl.7 ................................................ G061“ 15716
`U.S. Cl.
`7097230; 709931; 3457327;
`3707259
`Field of Search ................................. 7097231, 230;
`3457327; 3707259
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,873,777
`3,932,698
`5,319,707
`5,448,568
`5,497,373
`5,544,198
`5,778,187
`5,784,277
`5,799,150
`5,302,294
`
`5,818,441>>>>>>3¥b>>b
`
`*
`
`37' 1975
`171976
`671994
`971995
`371996
`371996
`771998
`77’ 1998
`871998
`971998
`1071998
`
`Uehara el al.
`
`Yanagimaehi el al.
`Wasilewski et al.
`Delpuch et al.
`Hulen et al.
`Saalfrank
`Monteiro el al.
`Meyer
`Ilamilton el al.
`Ludwig et al.
`Throckmorlon el al.
`
`3707259
`
`[171998 Davis at al.
`5,838,678 A
`671999 Kouloheris et al.
`5,915,094 A *
`7.71999 Goetz el al.
`5,928,330 A
`971999 Goetz el al.
`5,956,729 A
`5,966,120 A * 1071999 Arazi et al.
`5,987,501 A
`11.71999
`Ilamilton et al.
`0,055,246 A
`472000 Jones
`6,134,243 A * 1072000 Jones et al.
`6,157,674 A * 1272(100 Oda el al.
`6,175,871 Bl "
`172001 Sehuster et al.
`
`345.3327
`
`345.3327
`
`3707465
`375.3240
`709723]
`FOREIGN PATENT DOCUMENTS
`
`EP
`I‘LP
`
`[1497449
`0702309
`
`871992
`371996
`
`OTHER PUBLICATIONS
`
`P. England, et al. “RAVE: Real—Time services for the Web,”
`Cotttpater Networks and ISDN Systettts 28 {1996), pp.
`1547—1558.
`A. Walsh, “Programming QuickTime, Multimedia to the
`Macs,"Dr. Dobbslottrtrm’, 17:7 (Jul. 1992), XPUtHlfitHBtB,
`pp. 76, 78—80, 102, and 104—105.
`PCT International Search Report for PCT Int’l Appln. No.
`PC’l‘tUS99ttKi953 mailed Jul. 26, 1999.
`PCT International Search Report for PCT Iut’l Appln. No.
`PC'17U5997’UU954 mailed Jul. 26, 1999.
`PCT International Search Report for PCT Int‘l Appln. No.
`PCT7'US99700955 mailed Jul. 26, 1999.
`
`* cited by examiner
`
`I’rttttrtry E'Jmtttr'ner—Krisua I.im
`(74) Attorney, Agent, or Ftrttt—Blakely, Sokoloff, Taylor &
`Zafman LLP
`
`{57)
`
`ABSTRACT
`
`Methods and apparatuses [or processing media data trans-
`mitted in a data communication medium. A digital process-
`ing system is provided with a time related sequence of media
`data provided to the digital processing system based on a set
`of data, wherein the set of data indicates a method to
`transmit the time related sequence of media data according
`to a transmission protocol. The set of data, itself, is a time
`related sequence of data associated with the time related
`sequence of media data. The time related sequence of media
`data may be presented andhtr stored by the digital process-
`ing system.
`
`44 Claims, 14 Drawing Streets
`
`{390
`or; i same «in H}: mm
`
`m
`
`'—_1:
`Deviance CliifiED massed worms:
`
`
`
`
`
`I
`
`UlEAYEi-tfl snare mrs‘ wermmnmrméim nap
`mosaics Lem mum yaw nuns mum-mu;
`Fonmamtsseu moo-row FE nanEmmsusm:
`Prflfflocu t'I7IT5 “1|:me SIWE is A WEI W “E
`I'ILll'LD mE-‘EE 6F DIME 71“th Fififi 10011111 was CF
`UEMIMHI
`l
`tumor! DATA Foil a mm; svsrL-a HIM ms
`MEN Ems: limit-T510 “Emacs 575m: L'i u r.‘
`
`rrwtsum‘m Br new n'r-nuw “For: museumone. u:
`hi PJN'IS
`
`r-régfin, Al or FSEFH‘M': 57579.1 are meant
`EFnEsEIr‘Fu.m of mu mu
`
`
`OPIWILY WHIL NEDA RI: 0'“ WW“ ”511'! Fll F
`"mu PROM HICKH S C: REM BAH I'IEEA'JVEG AI TIE.
`
`BEBE“!!! SVSYUJ
`
`
`3:5
`
`Apple Exhibit 4213
`
`Apple v. SightSound Technologies
`CBM2013-00020
`
`Page 00001
`
`Apple Exhibit 4213
`Apple v. SightSound Technologies
`CBM2013-00020
`Page 00001
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 1 of 14
`
`US 6,453,355 B1
`
`eader
`
`II
`
`'de
`
`
`
`media data
`
`U
`
`D D
`
`El El
`
`El El
`
`El
`
`.
`
`chunk Shame: Etrameg
`El
`El El
`El El
`
`Shame:
`El El
`El
`
`- - -
`
`FIG. 1
`
`Page 00002
`
`Page 00002
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 2 0f 14
`
`US 6,453,355 B1
`
`movie header
`
`II
`
`(video) track
`
`
`
`()audio track—-—-_
`
`III!!!”
`
`—-.-__._.-
`El U
`D U
`m
`El ElEl I
`
`. chunk?ElrameEl33.5.
`
`
`
`u i EdchunkElam: -amefl firlrMEIl'- -| U El U
`
`FIG. 2
`
`Page 00003
`
`Page 00003
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 3 of 14
`
`US 6,453,355 Bl
`
`/’ 300
`
`DETERMINE MEDIA FILE FORMAT
`
`DETERMINE DESIRED TRANSMISSION PROTOCOL(S)
`
`CREATE AND STORE "HINTS" FOR PACKETIZING A TIME RELATED
`SEQUENCE OF MEDIA DATA IN A MEDIA FILE(S) (PACKETIZING IS
`FOR TRANSMISSION ACCORDING TO THE DESIRED TRANSMISSION
`PROTOCOL) (HINTS ARE NORMALLY STORED AS A TRACK OF TIME
`RELATED SEQUENCE OF HINTS WHICH REFER TO OTHER TRACKS OF
`
`MEDIA DATA)
`
`
`TRANSMIT DATA FROM A TRANSMITTING SYSTEM HAVING THE
`MEDIA FILE(S) AND HINTS TO A RECEIVING SYSTEM; DATA IS
`TRANSMITTED BY PACKETIZING THE MEDIA DATA ACCORDING TO
`THE HINTS
`
`PRESENT, AT THE RECEIVING SYSTEM, THE MEDIA OBJECT
`REPRESENTED BY THE MEDIA DATA
`
`
`
`OPTIONALLY REASSEMBLE MEDIA FILE (IN ORIGINAL MEDIA FILE
`FORMAT) FROM PACKETS OF MEDIA DATA RECEIVED AT THE
`RECEIVING SYSTEM
`
`
`
`301
`
`303
`
`305
`
`307
`
`309
`
`311
`
`FIG. 3
`
`Page 00004
`
`Page 00004
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 4 0f 14
`
`US 6,453,355 Bl
`
`movie heade
`
`
`(video) track
`
`trak
`
`RTP hint
`
`track
`
`_—-
`
`fl
`
`
`
`_V--
`
`RTP Pawelhinl
`
`-
`I
`
`u
`III
`
`
`
`-
`
`:-
`
`.
`
`_ I
`
`HTP Packet hint
`
`I530”:
`:' W01?
`
`
`
`
`II
`
`
`
`
`
`If.”
`
`
`
`
`
`
`-
`:l .
`
`-j
`. I
`
`
`
`RTP Packet him
`.
`
`IE
`
`
`
`405
`
`FIG. 4
`
`Page 00005
`
`Page 00005
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 5 of 14
`
`US 6,453,355 Bl
`
` Simple Movie—Video only
`
`--Videometa-denim}'Pmeta-datahamfli
`
`
`——-T—_—‘
`mdaI
`sampiel
`salmple
`FTP-13mm“
`'a_PPckeIhim|
`_TPPackeIhim|
`.IaIInlerl
`Hui—marl
`“Mun—er!
`
`
`
`
`fit—_____
`-——-———
`
`
`
`mp mm track ———_-
`l—_-
`
`
`
`
`
`E
`
`Page 00006
`
`Page 00006
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 6 of 14
`
`US 6,453,355 Bl
`
`602
`
`622
`
`CUENT
`COMPUTER
`
`SYSTEM
`
`
`
`
` INTERNET
`
`CUENT
`COMPUTER
`
`
`SYSTEM
`VVEB
`
`
`SERVER
`
`SYSTEM
`
`610
`
`61
`
`NEWNORK
`
`INTERFACE
`
`CUENT
`
`COMPUTER
`SYSTEM
`
`
`
`
`COMPUTER
`
`SYSTEM
`
`
`
`618
`620
`
`CUENT
`
`
`
`FIG. 6
`
`Page 00007
`
`Page 00007
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 7 of 14
`
`US 6,453,355 Bl
`
` 652
`
`
`PROCESSOR
`
`MEMORY
`
`654
`
`656
`
`—_
`
`
`
`658
`
`
`
`DISPLAY
`CONTROLLER
`
`MASS
`MEMORY
`
`IIO
`CONTROLLER
`
`664
`
`662
`
`
`
`DISPLAY
`
`660
`
`IIO
`DEVICES
`
`666
`
`
`
`DIGITAL PROCESSING
`SYSTEM
`
`@
`
`MODEM
`
`OR
`
`NETWORK
`
`INTERFACE
`
`668
`
`FIG. 7
`
`Page 00008
`
`Page 00008
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 8 of 14
`
`US 6,453,355 Bl
`
`
`
`MEDIA
`PROCESSING
`
`
`
`UNIT
`
`
`
`
`
`@
`LINK 636
`
`
`
`684
`
`688
`
`HINT GENERATION
`AND PROCESSING UNIT
`
`590
`
`MEDIA PROCESSING
`UNIT
`
`692
`
`DATA COMMUNICATION
`
`UNIT
`
`
`
`
`
`
`
`
`
`
`
`
`
`CLIENT DATA
`
`DATA
`
`SYSTEM 680 /
`
`FIG. 8
`
`SERVER 6_9_1_I
`
`Page 00009
`
`Page 00009
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 9 of 14
`
`US 6,453,355 Bl
`
`MEDIA
`PROCESSING
`
`UNIT
`
`
`
`
`
`
`684
`CLIENT DATA
`DATA
`
`PROCESSING SYSTEM
`COMMUN'CAT'ON
`682
`
`
`—
`LINK 586
`
`
`702
`
`HINT PROCESSING
`
`UNIT
`
`704
`
`MEDIA PROCESSING
`
`UNIT
`
`705
`DATA COMMUNICATION
`UNIT
`
`
`
`
`
`
`
`
`
`SERVER m
`
`
`
`DATA
`
`SYSTEM 696 /
`
`COMMUNICATION
`LINK 7’08
`
`
`
`
`712
`
`HINT GENERATION
`
`UNIT
`
`GENERATOR m
`
`FIG. 9
`
`Page 00010
`
`Page 00010
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 10 0f14
`
`US 6,453,355 B1
`
`DETERMINE MEDIA FORMAT (IF MORE THAN
`ONE FORMAT IS/WILI. BE USED)
`
`720
`
`DETERMINE DESIRED DATA COMMUNICATION
`PROTOCOL(S) (IF MORE THAN ONE PROTOCOL
`ISIWILL BE USED)
`
`722
`
`724
`
`BASED ON THE MEDIA FORMAT AND DESIRED
`DATA COMMUNICATION PROTOCOL(S), CREATE
`AND STORE HINTS FOR PACKETIZING A
`TIME RELATED SEQUENCE OF MEDIA DATA
`
`OPTIONALLY TRANSMIT HINTS TO ANOTHER
`DIGITAL PROCESSING SYSTEM
`
`726
`
`FIG. 10
`
`Page 00011
`
`Page 00011
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 11 of 14
`
`US 6,453,355 B1
`
`RECEIVE, AT A RECEIVING SYSTEM,
`MEDIA DATA TRANSMITI'ED TO THE
`
`WITH "HINTS"
`
`RECEIVING SYSTEM IN ACCORDANCE
`
`730
`
`PRESENT, AT THE RECEIVING SYSTEM,
`A MEDIA OBJECT REPRESENTED BY
`
`THE MEDIA DATA
`
`732
`
`OPTIONALLY STORE THE MEDIA DATA
`AS A MEDIA FILE
`
`734
`
`OPTIONALLY REASSEMBLE THE STORED
`
`736
`
`MEDIA FILE
`
`FIG. 11
`
`Page 00012
`
`Page 00012
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 12 0f14
`
`US 6,453,355 B1
`
`GENERATOR
`OS
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`HINT CREATION
`ROUTINE(S)
`
`CREATED HINTS
`
`INFORMATION RELATING
`
`TO ONE OR MORE DATA
`COMMUNICATION PROTOCOLS
`AND/OR MEDIA FORMATS
`
`MACHINE-READABLE
`
`STORAGE MEDIUM
`
`740
`
`FIG. 12
`
`
`m
`
`SERVER OS
`
`NETWORK TRANSMISSION
`ROUTINE(S)
`
`MEDIA DATA
`
`MEDIA PACKETIZATION
`ROUTINE(S), BASED
`ON HINTS
`
`MACHINE-READABLE
`
`STORAGE MEDIUM
`
`FIG. 13
`
`Page 00013
`
`Page 00013
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 13 0f 14
`
`US 6,453,355 Bl
`
`772
`
`773
`
`RECEIVING SYSTEM
`OS
`
`MEDIA PRESENTATION
`ROUTINE(S)
`
`(OPTIONAL)
`MEDIA DATA
`
`NETWORK TRANSMISSION
`FIOUTINE(S)
`
`732
`
`fl
`
`774
`
`776
`
`IIAOEIEI-IIODNSIA
`FIE-ASSEMBLY ROUTINE
`
`MACHINE-READABLE
`
`STORAGE MEDIUM
`
`FIG. 14
`
`Page 00014
`
`Page 00014
`
`

`

`US. Patent
`
`Sep.17,2002
`
`Sheet 14 0f14
`
`US 6,453,355 B1
`
`HINT PACKET
`
`MEDIA DATA PACKET
`
`w
`
`DATA STORAGE AND/OR
`COMMUNICATION MEDIUM
`
`FIG. 15
`
`Page 00015
`
`Page 00015
`
`

`

`US 6,453,355 Bl
`
`2
`
`and temporal mapping information adjusted. When edits are
`complete, the relevant media data and meta—data may be
`rewritten into a single, interleaved, and optimized file for
`local or network access. Both the structured and the opti-
`mized files are valid QuickTime files, and both may be
`inspected, played, and reworked.
`The use of structured ("non—fiat”) files enables the same
`basic media data to be used and re-used in any number of
`presentations. This same advantage applies when serving, as
`will be seen below.
`
`In both editing and serving, this also permits a number of
`other files to be treated as part of a movie without copying
`the media data. Thus editing and serving may be done
`directly frOm files such as Sun Microsystem’s “au” audio
`format or the AVI video format, greatly extending the utility
`of these formats.
`
`The QuickTime file is divided into a set of objects, called
`atoms. Each object starts with an atom header, which
`declares its size and type:
`
`class Alom{
`int(32)
`char
`byte
`
`size:
`lypcI4I:
`onnlents|
`
`I:
`
`The size is in bytes, including the size and type header
`fields. The type field is four characters (usually printable), to
`permit easy documentation and identification. The data in an
`object after the type field may be fields, a sequence of
`contained objects, or both.
`A file therefore is simply a sequence of objecLs:
`class File {
`Atom U;
`
`}T
`
`he two important top—level objects are the media—data
`(mdat) and the meta-data (moov).
`The media-data objecl(s) contain the actual media (for
`example, sequences of sound samples). Their format is not
`constrained by the file format; they are not usually objects.
`Their format
`is described in the meta-data, not by any
`declarations physically contiguous with them. So,
`for
`example,
`in a movie consisting solely of motion-JPEG,
`JPEG frames are stored contiguously in the media data with
`no intervening extra headers. The media data within the
`media data objects is
`logically divided into chunks;
`however, there are no eXplicit chunk markers within the
`media data.
`When the QuickTime file references media data in other
`files, it is not required that these ‘secondary’ files be for—
`matted according to the QuickTime specification, since such
`media data files may be formatted as if they were the
`contents of a media object. Since the QuickTime format does
`not necessarily require any headers or other information
`physically contiguous with the media data, it is possible for
`the media data to be files which contain ‘foreign’ headers
`(e.g. UNIX “.au" files, or AVI files) and for the Quick'l'ime
`meta—data to contain the appropriate declarative information
`and reference the media data in the 'foreign’ file. In this way
`the QuickTime file format can be used to update, without
`copying, existing bodies of material in disparate fomiats.
`The QuickTime file format is both an established format and
`is able to work with, include, and thereby bring forward,
`other established formats.
`
`[free space (e.g. deleted by an editing operation) can also
`be described by an object. Software reading a file that
`
`Page 00016
`
`1
`METHOD AND APPARATUS FOR MEDIA
`DATA TRANSMISSION
`
`1!]
`
`15
`
`1E]
`
`40
`
`This application claims the benefit of the filing date of
`US. Provisional Patent Application Ser. No. 60;“071,566,
`flled Jan. 15, 1998.
`
`FIELD OF THE INVENTION
`
`The present invention relates to methods and apparatuses
`for preparing time related sequences of media data for
`transmission, and more particularly to packetired transmis-
`sion of such media data.
`
`INTRODUCTION AND BACKGROUND
`
`There are various different file structures used today to
`store tirne—based media: audio formats such as AIFF, video
`formats such as AVI, and streaming formats such as Real—
`Media. One reason that such file structures are different is
`their dilferent focus and applicability. Some of these formats
`are sufficiently relatively widely accepted, broad in their
`application, and somewhat simple to implement, and thus,
`may be used not onlyr
`for content delivery but also as
`interchange formats. Foremost among these general formats
`is the QuickTime file format. It is used today in the majority
`of web sites serving time—based data;
`in the majority of
`authoring environments, including professional ones; and on
`the majority of multimedia (TDROM titles.
`The QuickTime media layer supports the efficient display
`and management of general multimedia data, with an
`emphasis on time—based material (video, audio, etc.). The
`media layer uses the QuickTime file format as the storage
`and interchange format for media information. The archi-
`tectural capabilities of the layer are generally broader than
`the existing implementations, and the file format is capable
`of representing more information than is currently
`demanded by the existing QuickTime implementations.
`In contrast to formats such as AVI, which were generally
`designed to support local random access of synchronized
`media, QuickTime alloWs systems to manage the data,
`relationships and timing of a general multimedia presenta—
`tion. In particular, the QuickTime file format has structures
`to represent the temporal behavior of general time—based
`streams, a concept which covers the time-based emission of
`network packeLs, as well as the lime-based local presentation
`of multimedia data.
`
`The existing QuickTime file format is publicly described
`by Apple Computer
`in the May 1996 File format
`specification, which may be found at the Quick'lime site,
`<http:;’x’.www.apple.comi'quicktime>.
`One aspect of the Quick’Iime file format is the concept
`that the physical structure of media data (the layout in disk
`records) is independent of, and described by, a logical
`structure for the file. The file is fully described by a set of
`”movie” meta-data. This meta-data provides declarative,
`structural and temporal infon'nalion about the actual media
`data.
`
`The media data may be in the same file as the description
`data, {the “movie" meta-data), or in other file(s). A movie
`structured into one file is commonly called “flat”, and is
`self-contained. Non-[lat movies can be stmctured to refer-
`ence some, or all, of the media data in other files.
`As such, the format is generally suited for optimization in
`different applications. For example, when editing
`(oompositing), data need not be rewritten as edits are applied
`and media is re-ordered; the meta-data file may be extended
`
`60
`
`Page 00016
`
`

`

`US 6,453,355 Bl
`
`3
`includes free space objects should ignore such free space
`objects, as well as objects at any level which it does not
`understand. This permits extension of the file at virtually any
`level by introducing new objects.
`The primary metaxlata is the movie object. A QuickTime
`file has exactly one movie object which is typically at the
`beginning or end of the file, to permit its easy location:
`
`4
`
`sampletable{
`size;
`lnt(32',1
`type[4] = ‘stbl';
`char
`sampledescription sd:
`lirnclosarnplt:
`tls:
`syncsampletable
`syncs:
`samplclocl'lunk
`sloc;
`sampler-tire
`ssim:
`chunkofiset
`coffsel:
`shadowsync
`ssync:
`
`class
`
`}
`
`class Movie {
`int(32)
`char
`MovieIIeader
`oontents
`
`}
`
`size:
`Ly'pcI-‘l-I = ‘moov’;
`nih:
`Atom[ ]:
`
`The movie header provides basic information about the
`overall presentation (its creation date, overall timescale, and
`so on). In the sequence ofoontained objects there is typically
`at least one track, which describes temporally presented
`data.
`
`class Traek{
`int(33)
`char
`Tmcklieader
`Cnl'tlcnls
`
`l
`
`sixe;
`type[4] = 'trak';
`th:
`AtomI
`
`I:
`
`The track header provides relatively basic information
`about
`the track (its II),
`timescale, and so on). Objects
`contained in the track might be references to other tracks
`(e.g. for complex compositing), or edit lists. In this sequence
`of contained objects there may be a media object, which
`describes the media which is presented when the track is
`played.
`The media object contains declarations relating to the
`presentation required by the track {e.g.
`that
`it is sampled
`audio, or MIDI, or orientation information for a 3|)scene).
`The type of track is declared by its handler:
`
`1!]
`
`15
`
`1E]
`
`40
`
`handler {
`inl(32]
`char
`int(8)
`hil(24J
`char
`char
`
`char
`bil(32)
`bil(32)
`string
`
`class
`
`audio
`
`}
`
`size:
`type[4] = 'hdlr':
`version:
`flags;
`handlerlype[4]:
`handlersubtypeH]
`
`manufacturerH]:
`handlcrflags;
`handlerflagsmask:
`componenlnamc:
`
`-— mhlr for media handlers
`v— vidc for video, snun for
`
`Within the media information there is likewise a handler
`
`60
`
`declaration for the data handler (which fetches media data),
`and a data information declaration, which defines which files
`contain the media data for the associated track. By using this
`declaration, movies may be built which span several files.
`At the lowest level, a sample table is used which relates
`the temporal aspect of the track to the data stored in the file:
`
`The sample description contains information about the
`media {e.g.
`the compression formats used in video). The
`time—to—sample table relates time in the track, to the sample
`{by index) which should be displayed at that time. The sync
`sample table declares which of these are sync (key) samples,
`not dependent on other samples.
`The sample—to—chunk object declares how to find the
`media data for a given sample, and its description given its
`index:
`
`sampletoclmnk {
`i nt(31}
`sine;
`char
`type[4] = ‘stsc':
`int(3'_l
`version;
`biLs(24,1
`flags:
`int(3"‘.-.)
`entryeount:
`for (int i=0; i-ccnlrycount; i++j {
`int(32'}
`firslchunk;
`int(32)
`samplesperchunk:
`int(32)
`sampchcscriplionindcx:
`
`l
`
`class
`
`l
`
`The sample size table indicates the size of each sample.
`The chunkofl'set table indicates the ofiset into the containing
`file of the start of each chunk.
`Walking the above-described structure to find the appro-
`priate data to display for
`a given time is
`fairly
`straightforward, generally involving indexing and adding.
`Using the sync table, it is also possible to back-up to the
`preceding sync sample, and roll forward ‘silently’ accumu-
`lating deltas to a desired starting point.
`FIG. 1 shows the structure of a simple movie with one
`track. Asimilar diagram may be found in the QuickTime file
`format documentation, along with a detailed description of
`the fields of the various objects. QuickTime atoms (objects)
`are shown here with their type in a grey box, and a
`descriptive name above. This movie contains a single video
`track. The frames of video are in the same file, in a single
`chunk ofdata. It should be noted that the ‘chunk’ is a logical
`construct only;
`it
`is not an object. Inside the chunk are
`frames of video, typically stored in their native form. There
`are no required headers or fields in the video frames them-
`selves.
`FIG. 2 is a diagram of a selfcontained file with both an
`audio and a video track. Fewer of the atoms are shown here,
`for brevity; the pointers from the tracks into the media data
`are, of course, the usual sample table declarations, which
`include timing information.
`The Quick'l'ime file format has a number of advantages,
`including:
`1) Scalability for size and bit—rates. The meta data is
`flexible, yet compact. This makes it suitable for small
`downloaded movies (e.g. on the Internet) as well as
`providing the basis for a number of high-end editing
`SySIClTIS.
`
`Page 00017
`
`Page 00017
`
`

`

`US 6,453,355 Bl
`
`5
`2) Physical structure is independent of the logical and
`temporal structure. This makes it possible to optimize
`the physical structure differently depending on the use
`the lie will have. In particular, it means that a single file
`format is suitable for authoring and editing; download-
`ing or placing on CDROMs; and for streaming.
`3) The file format has proven capable of handling a very
`broad variety ol‘ oodec types and lrack types, including
`many not known at the time the format Was designed.
`This proven ability to evolve in an upwards—compatible
`fashion is fundamental
`to the success of a storage
`format.
`Scalable, or layered, codecs can be handled in a number
`of ways in the QuiekTirne file format. For a streaming
`protocol which supports scalability,
`the samples may be
`tagged with the layer or bandwidth threshold to be met for
`transmitting the samples.
`'l‘racks which form a set of alternatives (e.g. dilTerent
`natural language sound tracks) can be tagged so that only
`one is selected for playback. The same structure can be used
`to select alternatives for streaming (e.g.
`for
`language
`selection). This capability is described in further detail in the
`QuickTime file format.
`When QuickTime displays a movie or track, the appro—
`priate media handler accesses the media data for a particular
`time. The media handler must correctly interpret the data
`stream to retrieve the requested data. For example, with
`respect to video media, the media handler typically traverses
`several atoms to find the location and size of a sample for a
`given media time. The media handler may perform the
`following:
`1. Determine the lime in the media time coordinate
`system.
`2. Examine the time—to—sample atom to determine the
`sample number that contains the data for the specified
`time.
`
`1!]
`
`15
`
`1E]
`
`3. Scan the sample-to-chunk atom to discover which
`chunk contains the sample in question.
`4. Extract the offset to the chunk from the chunk ofi'set
`atom.
`
`40
`
`5. Find the otfset within the chunk and the sample’s size
`by using the sample size atom.
`It is often desirable to transmit a QuickTime file or other
`types of time related sequences of media data over a data
`communication medium, which may be associated with a
`computer network (e.g.
`the Internet). In many computer
`networks,
`the data which is transmitted into the network
`should generally be in a packet form. Normally, time related
`sequences of media data are not
`in the proper packetized
`format for transmission over a network. For example, media
`data files in the QuiekTime format are not in a packetized
`format. Thus, there exists a need to collect the data, some-
`times referred to as streaming data, into packets for trans-
`mission over a network.
`
`One prior approach to address the problem of transmitting
`time related sequences of media data over a network is to
`send the media file over the network using a network or
`transmission protocol, such as the Hypertext Transfer Pro-
`tocol (HTTP). Thus, the media file itself is sent from one
`computer system over the network to another computer
`system. However, there may be no desire to retain the media
`file at the receiving computing system. That is, when the
`media file is received and viewed or listened to at
`the
`
`60
`
`receiving computer system, there may be no desire by the
`user of that receiving computer system to store a copy of the
`file, for example, if the receiving computing system is a
`network computer or a computer with low storage capacity.
`
`6
`Another alternative approach to solving the problem of
`how to collect data for transmission by packets over a
`network is to prepare a file which contains the network
`protocol data units in the file for a particular transmission
`protocol.
`In a sense, such a file may be considered a
`packetized file which is stored in essentially the same format
`as it will be transmitted according to the particular trans—
`mission protocol. Performing this operation generally
`involves storing the file in a packetized form for a particular
`network protocol at a particular data transmission rate and a
`particular media file format. Thus, for each diIferent trans-
`mission protocol at a particular data transmission rate, the
`file will essentially be replicated in its packetized form. The
`fixed form of such files may restrict
`their applicability;{
`compatibility and make it diflicult to view such files locally.
`Thus, such an approach may greatly increase storage
`requirements in attempting to provide the file in various
`transmission protocols at various ditferent data transmission
`rates. Moreover, each packetimed file generated according to
`this alternative prior approach is generally limited to a
`particular media file format, and thus, other media file
`formats for the same media object (e.g. a digital movie) are
`typically packetized and stored on the sending computer
`system.
`Yet another approach to solving the problem of how to
`stream time related sequences of media data is to perform
`the packetization of the media data when required on the
`transmitting system according to the particular transmission
`promcol which is desired. This processing requires, in many
`cases, a relatively considerable amount of time, and thus,
`may slow the performance of the transmitting system.
`Thus, it is desirable to provide an improved method and
`apparatus for transmitting time related sequences of media
`data.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides methods and apparatuses
`for processing media data transmitted in a data communi—
`cation medium. In one embodiment, a digital pmceSsing
`system receives a
`time related sequence ol‘ media data
`provided to the digital processing system based on a set of
`data, wherein the set of data indicates a method to transmit
`the time related sequence of media data according to a
`transmission protocol. The set of data, iLself, is a time related
`sequence of data associated with the time related sequence
`of media data. The time related sequence of media data may
`be presented andtor stored by the digital processing system.
`
`BRIEF DESCRIPTION OI" ’I'IIL" DRAWINGS
`
`FIG. 1 shoWs an example of the structure of a simple
`movie with one track in the prior art.
`FIG. 2 is an example of a self-contained movie file of the
`prior art.
`FIG. 3 is a flowchart showing one example of a method
`according to the present invention.
`FIG. 4 shows an example of a hint track of the present
`invention.
`
`track of the
`
`FIG. 5 shows another example of a hint
`present invention.
`FIG. 6 is a diagram of a network of computer systems in
`which media data may be exchanged andi'or processed,
`according to one embodiment of the present invention.
`FIG. 7 is a block diagram of a digital processing system
`which may be used in accordance with one embodiment of
`the present invention.
`
`Page 00018
`
`Page 00018
`
`

`

`US 6,453,355 Bl
`
`7
`FIG. 8 is a block diagram of a system that utilizes hints
`to transfer media data, according to one embodiment of the
`invention.
`
`FIG. 9 is a block diagram of a system that utilizes hints
`to transfer media data, according to one embodiment of the
`invention.
`
`FIG. 10 is a flow diagram illustrating a method for
`generating hints for providing media data transmission,
`according to one embodiment of the invention.
`FIG. 11 is a flow diagram illustrating a method of pro—
`cessing media data received by a receiving system in accor—
`dance with hints, according to one embodiment of the
`invention.
`
`FIG. 12 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a generator, according to one embodiment of
`the invention.
`
`['10. 13 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a server, according to one embodiment of the
`invention.
`
`FIG. 14 is an example of a machine readable storage
`medium that may be accessed by a digital processing
`system, such as a receiving system or other digital process-
`ing system, according to one embodiment of the invention.
`FIG. 15 is a diagram of a data storage andtbr commu ni—
`cation medium having storeditransported thereon media and
`hint information, according to one embodiment of the inven—
`1i()I].
`
`DETAILED DESCRIPTION
`
`The present invention provides methods and apparatuses
`for allowing the transmission, and particularly the pack—
`etimed transmission of time related-sequences of media data,
`which may include, for example, video, audio, video and
`audio, etc., over a communication media, such as in a
`computer network.
`In one embodiment of the present invention, a digital
`processing system creates a set of data for indicating how to
`transmit a time related sequence of media data according to
`a transmission protocol. Typically, this set of data is stored
`on a storage device coupled to the digital processing system.
`Further, this set of data is a time related sequence of data
`associated with the time related sequence of media data.
`The present invention may be implemented entirely in
`executable computer program instructions which are stored
`on a computer readable media or may be implemented in a
`combination of software and hardware, or in certain
`embodiments, entirely in hardware. Typically, a server com-
`puter system coupled to a network will create the set of data,
`which may be referred to as a hint track and will store this
`hint track in a storage device which is coupled to the server
`computer system. When a client computer system requests a
`presentation (e.g. a viewing or listening or viewing and
`listening)t1fa media data file, the server system uses the hint
`track to determine how to packetize the media data for
`transmission to the client computer system. It will be appre—
`ciated that the present invention is generally applicable to
`time related sequences of media data, and that Quick'l'ime is
`represented herein as one example of this general applica-
`bility. Thus, the invention should not necessarily be limited
`to QuickTime.
`FIG. 3 shows one example of a method according to the
`present invention. The method 300 shown in FIG. 3 begins
`in step 301, in which the media file format for the particular
`
`It]
`
`‘15
`
`1t]
`
`40
`
`60
`
`8
`media data which is desired to be transmitted is determined.
`In step 303, the particular transmission protocol or protocols
`which are desired to be used is also determined. However,
`steps 301 and 303 are optional, for example,
`in the case
`where the same media file format
`is always transmitted
`using the same transmission protocol.
`In step 305, a digital processing system, such as a server
`computer system, creates and stores the hints for packetizing
`a time related sequence of media data in a media file.
`Alternatively, one computer system may create the hints and
`provide them to another system, such as a server computer
`system, which stores them for later use in a transmission
`process. The packetitration allows the transmission over a
`network or communication media according to the desired
`transmission protocol which was determined in step 303. In
`one embodiment of the present
`invention,
`the hints are
`stored as a track of time related sequence of hints which
`refers to, but which in one embodiment, is separate from
`other tracks of media data. The track of hints,
`in one
`embodiment of the present invention, may be stored sepa-
`rately from the media data to which it refers. As such, the
`track of hints may be stored in a file which is distinct from
`another file containing the media data which is referred to by
`the track of hints, or the track of hints may be stored in a hint
`area in the file containing the media data which is separate
`and distinct from the data area containing the actual media
`data. In one embodiment of the invention, a hint track, or
`portion thereof, may be interpreted as executable instmc-
`tions by the server, which executable instructions cause the
`server to packetize a time related sequence of data, which is
`typically, but not necessarily, time—based media data. In one
`embodiment of the present invention, the hints are stored on
`the storage device which is coupled to the transmitting
`digital processing system.
`In step 307, the data which is packelized according to the
`hints, is transmitted from a transmitting system, such as a
`server