USOO7653685B2
`
`(12) United States Patent
`US 7,653,685 B2
`(10) Patent N0.:
`
`(45) Date of Patent: *Jan. 26, 2010
`Serenyi et al.
`
`(54)
`
`METHODS AND APPARATUSES FOR
`TRANSFERRING STREAMING
`MULTIMEDIA DATA AND VARIOUS TYPES
`OF DATA ASSOCIATED THEREWITH IN A
`NETWORK ENVIRONMENT
`
`FOREIGN PATENT DOCUMENTS
`
`WO
`
`97/22201 A2
`
`6/1997
`
`(75)
`
`Inventors: Denis Serenyi, Menlo Park, CA (US);
`Chris LeCroy, Mountain View, CA (US)
`
`(Continued)
`OTHER PUBLICATIONS
`
`(73)
`
`Assignee: Apple Inc., Cupertino, CA (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`Carsten Griwodz, Alex Jonas, Michael Zink, “Affordable Infrastruc-
`ture for Stream Playback in the Interne ”, Darmstadt University of
`Industrial Process and System Communications (KOM), Dec. 12,
`1999, Technical Report TR-KOM-l999-07, Darmstadt, Germany,
`(pp. 1-17)
`
`This patent is subject to a terminal dis-
`claimer.
`
`(Continued)
`
`(21)
`
`Appl. N0.: 11/355,805
`
`(22)
`
`Filed:
`
`Feb. 15, 2006
`
`Primary ExamineriRobert B Harrell
`(74) Attorney, Agent, or FirmiBlakely, Sokoloff, Taylor &
`Zafman LLP
`
`(65)
`
`(60)
`
`(51)
`
`(52)
`(58)
`
`(56)
`
`Prior Publication Data
`
`(57)
`
`ABSTRACT
`
`US 2006/0143303 A1
`
`Jun. 29, 2006
`
`Related US. Application Data
`
`Division ofapplication No. 09/757,235, filed on Jan. 8,
`2001, now Pat. No. 7,007,062, which is a continuation-
`in-part of application No. 09/603,108, filed on Jun. 22,
`2000, now Pat. No. 7,191,242.
`
`Int. Cl.
`
`(2006.01)
`G06F 13/00
`
`709/203
`US. Cl.
`.......................................
`Field of Classification Search .................. 709/203
`
`See application file for complete search history.
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,918,020 A
`
`6/1999 Blackardet a1.
`
`The present invention provides several methods and appara-
`tuses for transmitting multimedia data using streaming media
`protocols such as real-time transfer protocols (RTP) and real-
`time streaming protocols (RTSP) in a computer network envi-
`ronment. A request for RTP data is sent from the caching
`proxy server to the server. The request may be for one specific
`type of data or multiple unrelated types of data. The server
`responds to the request indicating its support for the requested
`data. The caching proxy server determines whether to pro-
`ceed or terminate the data transmission process based on the
`response provided by the server. If it is determined to proceed
`with the data transmission process, the caching proxy informs
`the server to send the requested data. The server sends the
`requested data in a body of a RTP packet. The RTP packet
`uses a RTP Meta-Info payload format, which includes a body
`and a field header. The field header includes fields to identify
`the streaming media data, and the field body includes the
`requested streaming media data.
`
`(Continued)
`
`33 Claims, 15 Drawing Sheets
`
`401
`
`CACHING PROXY SERVER (CPI
`
`‘
`
`TRANSMISSION/RECEIPT
`CONTROL SOFTWARE
`(EG. FOR RTSP AND RTP
`
`1102
`
`DATA 2
`
`STREAMING MEDIA
`DATA I
`(IN STREAMING
`MEDIA FORMAT)
`
`1T ASSOCIATED WITH
`STREAMING MEDIA
`DATA 1
`
`FT ASSOCIATED WITH
`STREAMING MEDIA
`DATA 1
`
`STREAMING MEDIA
`DATA 2
`(IN STREAMING
`MEDIA FORMAT)
`
`1TASSOCIATED WITH
`STREAMING MEDIA
`DATA 2
`
`Fl' ASSOCIATED WITH
`STREAMING MEDIA
`
`Apple Exhibit 4435
`
`Apple V. SightSound Technologies
`CBM2013-00023
`
`Page 00001
`
`Apple Exhibit 4435
`Apple v. SightSound Technologies
`CBM2013-00023
`Page 00001
`
`

`

`US 7,653,685 B2
`
`Page 2
`
`US. PATENT DOCUMENTS
`
`W0
`
`WO 00/27087 A1
`
`5/2000
`
`6,252,889 B1
`6,263,371 B1
`6,275,471 B1
`6,300,887 B1
`6,359,656 B1
`6 415 327 B1
`6,505,169 B1 *
`,
`,
`2523:5321; :1
`6,629,138 B1
`’
`’
`6’639’896 B1
`6’674’477 B1
`6,708,213 B1
`6,711,131 Bl
`6,771,644 B1
`
`6/2001 Patki et a1.
`7/2001 Geagan, 111 et al.
`8/2001 Bushmitch et 31,
`10/2001 Le
`3/2002 Huckins
`7/2002 Beckerman et al.
`1/2003 Bhagavath et al
`$388: WFynllltagtlet al.
`9/2003 L
`b t
`it
`1
`/
`amd er e la '
`10 2003 G00 e et a. '
`“2004 Yamaguchi et 31'
`3/2004 Bommaiah et a1.
`3/2004 Shiobara
`8/2004 Brassil et al.
`
`.
`
`705/14
`
`............
`
`FOREIGN PATENT DOCUMENTS
`
`W0
`W0
`WO
`
`WO 97/22201 A2
`WO 99/37057
`00/27087 A1
`
`6/1997
`7/1999
`5/2000
`
`OTHER PUBLICATIONS
`Henning Schulzrinne, Jonathan Rosenberg, “Internet Telephony:
`architecture and protocolsi an IETF perspective,” Computer Net-
`works, Elsevier Science Publishers: Amsterdam, NL, vol. 31 No. 3,
`Feb 1_1, 1999 (PR 237-255)
`_
`Henning Schulzrinne, et al. “RTP: A transport protocol for real-time
`applications,” Network Working Group RFC 1889 RTP, Jan. 1996
`(pp. 1-75).
`PCT International Search Report for PCT Int’l Appln No. USOl/
`20044, mailed May 22, 2002 (8 pages).
`Chinen, Ken-Ichi, et al., “An Interactive Prefetching Proxy Server for
`Improvement ofWW Latency”, downloaded from the internet Feb.
`1, 2008 from http://isoc.org/inet97/proceedings/A1/A173.HTM, 10
`pages.
`Civanlar, M. Reha “Protocols for Real-Time Multimedia Data Trans-
`mission Over the Internet”, Acoustics, Speech and Signal Processing,
`1998. Proceedings of the 1998 IEEE International Conference on
`Digital Object Identifier; 10.1109/CASSP 1998.679714 1 page bib-
`liography/summary page.
`Hoffman, D. et al. “RTP Payload Format for MPEG1/MPEG2
`Video,” Jan. 1998, RFC 2250.
`
`* cited by examiner
`
`Page 00002
`
`Page 00002
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 1 of 15
`
`US 7,653,685 B2
`
`ORIGINATING SERVER AND CACHING PROXY SERVER(CP)
`NEGOTIATE, THROUGH REAL-TIME STREAMING PROTOCOL
`(RSTP) FOR SERVER TO SEND AND OF SERVER TO RECEIVE
`STREAMING MEDIA DATA (FOR EXAMPLE: OUICKTIME
`STREAMING AUDIO DATA).
`
`CONTROLLED BY THE CP SERVER.
`
`OR SERVER RECEIVES STREAMING MEDIA DATA. USING REAL
`TIME TRANSFER PROTOCOL (RTP), IN STREAMING MEDIA
`FORMAT NOT THE SAME FORMATAS STORED ON ORIGINATING
`SERVER AND OP SERVER STORES THE STREAMING MEDIA DATA.
`IN THE STREAMING MEDIA FORMAT. ON A STORAGE DEVICE
`
`CP SERVER RECEIVES REQUEST. THROUGH RSTP. FROM A
`CLIENT FOR THE STREAMING DEDIA DATA AND RETRIEVES THE
`DATA FROM THE STORAGE DEVICE AND TRANSMITS THE
`STREAMING MEDIA DATA TO THE CLIENT SYSTEM.
`
`FIG. 1A
`
`(Prior Art)
`
`Page 00003
`
`Page 00003
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 2 of 15
`
`US 7,653,685 B2
`
`122
`
`N
`
`INTERNET
`
`
`
`
`132
`
`
`CP SET/En
`
`
`a
`
`
`100
`
`\«
`
`102
`
`106
`
`CLIENTTER m
`
`COMPU
`SYSTEM
`
`104
`
`cuem
`
`COMPUTER
`
`108
`
`110
`
`GATEWAY
`SYSTEM
`
`112
`
`125
`
`WEB
`SERVER
`
`SYSTEM
`
`1 14
`
`1 16
`
`NETWORK
`INTERFACE
`
`‘
`
`NETWORK
`INTERFACE
`
`118
`
`120
`
`cum
`COMPUTER
`svsmc
`
`‘rsm
`comma
`svsma
`
`FIG. 18
`
`Page 00004
`
`Page 00004
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 3 of 15
`
`US 7,653,685 B2
`
`
`
`
`
`252
`{\J
`
`255
`
`
`
`
`
`I
`
`258
`
`262
`
`
`
`I 264
`
`CONTROLLER
`
`MEMORY
`
`DISPLAY
`
`260
`
`CONTROLLER
`
`266
`
`IIO
`DEVICES
`
`
`
`
` DIGITAL PROCESSING
`
`SYSTEM 259
`
` MODEM OR
`
`NETWORK
`
`INTERFACE
`
`
`FIG. 2
`
`Page 00005
`
`Page 00005
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 4 of 15
`
`US 7,653,685 B2
`
`301
`
`SERVER(S)
`
`CLIENT(C)
`
`RTSP & RTP
`
`302
`‘
`
`FIG. 3
`
`301
`
`401
`
`302
`
`cusmrm)
`
`CACHING PROXY
`SERVER (op)
`
`scam/ems;
`
`RTSP 8: RTP
`
`FIG. 4
`
`Page 00006
`
`Page 00006
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 5 of 15
`
`US 7,653,685 B2
`
`
`
`CP ASKS SETUP IN RTSP
`FOR AUDIO DATA AND SPECIFIES
`EXTENSIONS TO RTP DAT
`FOR AUDIO DATA
`
`SERVER RESPONDS/NEGOTIATES
`
`OP ASKS SETUP IN RTSP
`FOR VIDEO DATA AND SPECIFIES
`
`EXTENSIONS TO RTP DATA
`
`SERVER RESPONDS/NEGOTIATES
`
`CP ASKS PLAY IN RTSP
`
`SERVER RESPONOS WITH RTP DATA
`FOR ALL TRACKS WITH NEGOTIATED
`
`EXTENSIONS
`
`501
`
`502
`
`503
`
`504
`
`505
`
`506
`
`FIG. 5
`
`Page 00007
`
`Page 00007
`
`

`

`U.S. Patent
`
`Jan. 26, 2010
`
`Sheet 6 of 15
`
`US 7,653,685 B2
`
`mom5?En
`
`:58“5E>$m
`
`HNVGE
`
`EEMESE
`
`m5:
`
`H._.<5259.<35
`
`mug—23:58mx85mcumF0mm5wow
`Sam.5.E.t.E.
`
`2515.25me255m255m
`
`
`
`xNFo<._.<a<55<._.<o<63Sc
`
`mommu2.wwmohm
`
`
`
`mExQE.203
`
`Page 00008
`
`Page 00008
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 7 of 15
`
`US 7,653,685 B2
`
` CP ASKS FOR TRANSMIT TIME
`
`INFORMATION FROM THE SERVER
`
`
`
` SERVER RECEIVES THE REQUEST FOR -----------------------------
`
`TRANSMIT TIME (IT)
`
`
`
`
`
`
`------------- .
`I
`;
`ASKEORAPEAQILSS
`I
`I
`I
`N. OTHER DATA DIRECTLY .
`711 ;
`FROM SERVER
`I
`----------- 'I' -*
`
`705
`
`
`SERVER DOES NOT RESPOND TO
`
`
`
`SERVER RESPONOS AND SENOS TT
`REQUEST. THIS INDICATES THAT
`
`IDENTIFIER CODE. WHICH
`SERVER DOES NOT SUPPORT THE
`
`
`
`CORRESPONDS. TO TT TYPE OF
`EXTENSION REQUESTED
`
`
`
`EXTENSION. IN AN EXTENSIBLE
`
`
`HEADER FORMAT - STREAMING DATA
`
`
`IS SENT WITH TT EXTENDED DATA
`
`
`CP MAY TERMINATE PROCESS
`
`
`IF TT IS CRITICAL ELEMENT
`
`
` -EXTENSIBLE HEADER FORMAT IS A
`OF THE CF OPERATION
`
`SUB‘EXTENSION CONTAINING A SUB-
`
`
`EXTENSION ID, IDENTIFYING THE
`CONTENT OF THE INFORMATION.
`-TI' SUB-EXTENSION ALSO
`CONTAINS A SINGLE 64-BIT
`
`
`UNSIGNED INTEGER REPRESENTING
`
`
`RECOMMENDED TRANSMISSION
`
`
`TIME OF RTP PACKET IN
`
`
`MILLISECONDS
`
`
` 705
` CP GETS THE TT DATA. REMOVES
`m“s’°”5‘c%°$"R’IIITE§KEEWE
`
`
`DATA L
`LL ;
`ASSOCIATED WITH APPROPRIATE
`
`STREAMING DATA
`
`
`
`
`CLIENT REQUESTS STREAMING DATA
`BY MAKING A PLAY REQUEST IN
`RTSP
`
`
` CLIENT PLAYS DATA
`
`-----------------------------
`
`Page 00009
`
`Page 00009
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 8 of 15
`
`US 7,653,685 B2
`
` CP ASKS FOR TRANSMIT TIME
`
`INFORMATION FROM THE SERVER
`
`
`
`
`
`
`
`
` SERVER RECEIVES THE REQUEST FOR -----------------------------
`FRAME TIME (FI')
`
`
`SERVER DOES NOT RESPOND TO
`
`
`SERVER RESPONDS AND SENDS FT
`
`REQUEST THIS INDICATES THAT
`
`
`IDENTIFIER CODE, WHICH
`SERVER DOES NOT SUPPORT THE
`
`
`
`CORRESPONDS, TO FT TYPE OF
`EXTENSION REQUESTED
`
`
`
`EXTENSION, IN AN EXTENSIBLE
`
`
`HEADER FORMAT - STREAMING DATA
`
`
`IS SENT WITH FT EXTENDED DATA
`
`
`CP MAY TERMINATE PROCESS
`
`808
`IF FT IS CRITICAL ELEMENT
`
`
`
`EXTENSIBLE HEADER FORMAT IS A
`OF THE CP OPERATION
`
`
`
`SUB-EXTENSION CONTAINING A SUB-
`
`
`EXTENSION ID, IDENTIFYING THE
`
`CONTENT OF THE INFORMATION.
`FT SUB-EXTENSION ALSO
`CONTAINS 64-BIT UNSIGNED INTEGER
`VALUE WITH WELL-KNOWN FRAME
`
`
`TYPES SUCH AS O-UNKOWN FRAME.
`
`
`1-KEY FRAME, 2-B FRAME, AND
`
`
`3-P FRAME. FUTURE VERSIONS
`
`CAN ADD MORE FRAMES.
`
`CLIENT INFORMS
`
`OR THAT IT IS
`
`DVERLOADED
`
`
`,- ---------- -'—-I
`WITH
`
`
`
`I
`CLIENT MAY ALSO
`;
`-
`INFORMATION
`
`
`
`;
`ASK FOR FI' OR
`.
`CP GETS THE Fr DATA. REMOVES
`
`EXTENSION HEADER AND STORES THE
`0 OTHER DATA DIRECTLY I
`
`
`310 I
`DATA LOCALLY; FTFRAMES ARE
`FROM SERVER
`I
`
`ASSOCIATED WITH APPROPRIATE
`'— .......... 1--.
` STREAMING DATA
`
` CP REVIEWS CLIENT REQUEST STOPS
`
`
`
`SENDING LEAST IMPORTANT FRAMES
`AND SENDS ONLY HIGHER
`IMPORTANCE FRAMES
`
`Page 00010
`
`Page 00010
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 9 of 15
`
`US 7,653,685 B2
`
`CP ASKS BY NAME FOR ONE OR MORE RTP
`SUB-EXTENSIONS USING RTSP PROTOCOL
`
`901
`
`
`
`SERVER RESPONDS BACK TO CP INDICATING ITS
`
` 903
`SUPPORT FOR THE REQUESTED SUB-EXTENSIONS.
`
`SERVER ALSO TRANSMITS TO CP AN IDENTIFIER (E.G..
`
`
`NUMBER CODE) CORRESPONDING TO EACH NAME
`
`
`OF RTP EXTENSION. CP MAY LATER USE THE CODE
`WHEN RECEIVING EXTENDED DATA. RESPONSE
`
`
`MAY OR MAY NOT BE A COMPLETE RESPONSE.
`
`
`CP RECEIVES SERVER'S RESPONSE
`905
`
`
`GP DETERMINES WHETHER SERVER RESPONDED
`TO ALL REQUESTED SUB-EXTENSIONS
`
`
`
` NO
`
`
`909
`
`
`YES
`919
`SERVER DID NOT RESPOND TO ALL
`SERVER RESPONDED TO ALL
`
`
`
`REQUESTED SUB-EXTENSIONS
`REQUESTED SUB'EXTENSIONS
`
`
`
`
` ARE ANY OF THE MISSING SUB-
`
`
`
`911
`CP REQUESTS SERVER TO SEND ALL
`EXTENSIONS CRITICAL TO CP
`SUPPORTED STREAMING MEDIA DATA
`
`
`
`
`PROCESSES?
`ASSOCIATED THE RTP SUB-EXTENSIONS
`
`
` YE
`
`
`I
`SERVER SENDS ALL SUPPORTED RTP SUB-
`S
`EXTENSIONS AND STREAMING MEDIA DATA
`
`
`ASSOCIATED WITH THE RTP SUB-
`CP DECIDES WHETHER
`
`EXTENSIONS
`TO TERMINATE OPERATIONS
`
`
`
`
`
`
`GP STORE SUB-EXTENSION DATA AND
`CP DECIDES TD PROCEEI
`
`STREAMING MEDIA ASSOCIATED WITH THE
`AND REQUEST THE
`
`
`
`RTP SUB~EXTENSION LOCALLY
`REMAINING/SUPPO RTED
`
`
`
`SUB-EXTENSIONS
`
`
`
`927
`
`C? MAY REMOVE SUB-EXTENSION FIELDS
`
`GP EVALUATES CLIENT REQUESTS AND
`PER FORMS FUNCTIONS ACCORDINGLY.
`FOR EXAMPLE FRAME THINNING OR USING
` 929
`
`TRANSMIT TIME OPSLONALLY TO TRANSMIT
`
`TA
`931
`
`
`917
`
`913
`
`CP
`TERMINATES
`OPERATIONS
`
`915
`
`
`
`FIG- 9
`
`Page 00011
`
`Page 00011
`
`

`

`U.S. Patent
`
`n.
`
`“J4
`
`m0m
`
`U
`
`mom
`
`9.mommzfioz
`
`
`
`mus—<52.x...
`
`.5539
`
`#65E
`
`t2m2<E
`
`m2:
`
`m,:3352%
`
`“5.;w225.52:mumEE".M“$on33mas2355
`
`“.23$5:mg}:
`
`2IwEa:w:38g8$25:m,mamy.
`:20
`582mm“5
`
`Page 00012
`
`Page 00012
`
`
`
`

`

`U.S. Patent
`
`Jan. 26, 2010
`
`Sheet 11 of 15
`
`US 7,653,685 B2
`
`401
`
`CACHING PROXY SERVER (CP)
`
`DATA 2
`
`TRANSMISSION/RECEIPT
`CONTROL SOFTWARE
`
`(E.G. FOR RTSP AND RTP
`
`STREAMING MEDIA
`DATA 2
`
`(IN STREAMING
`MEDIA FORMAT)
`
`TT ASSOCIATED WITH
`STREAMING MEDIA
`DATA 2
`
`FT ASSOCIATED WITH
`STREAMING MEDIA
`
`STREAMING MEDIA
`DATA 1
`(IN STREAMING
`MEDIA FORMAT)
`
`TT ASSOCIATED WITH
`STREAMING MEDIA
`DATA 1
`
`FT ASSOCIATED WITH
`STREAMING MEDIA
`DATA 1
`
`FIG. 11
`
`Page 00013
`
`Page 00013
`
`

`

`U.S. Patent
`
`Jan. 26, 2010
`
`Sheet 12 of 15
`
`US 7,653,685 B2
`
`301
`
`SERVER (ORIGINATING SERVER)
`
`TRANSCEIVER
`CONTROL SOFTWARE
`
`ORIGINAL DATA FILE
`FOR MEDIA
`
`(IN ORIGINAL
`DATA FO R MAT)
`
`RTP EXTENSIONS
`
`NAME
`TT
`FT
`
`CODE
`01
`02
`
`EXTENSIONS
`
`NM
`
`SOFTWARE FOR
`TRANSMISSION AND
`CONTROL OF RTP
`
`STREAMING MEDIA
`CONVERSION
`SOFTWARE (CONVERTS
`ORIGINAL FILE TO
`STREAMING FORMAT)
`
`FIG. 12
`
`Page 00014
`
`Page 00014
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 13 of 15
`
`US 7,653,685 B2
`
`302
`
`CLIENT
`
`STREAMING MEDIA
`DATA PLAYER
`SOFTWARE
`
`SOFTWARE
`
`OS
`
`1301
`
`WEB BROWSER
`
`‘303
`
`FIG. 13
`
`Page 00015
`
`Page 00015
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 14 of 15
`
`US 7,653,685 B2
`
`+—+—-+
`
`—+.......+
`
`I
`I I O
`-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+—+-+-+-+-+-+-+-+-+-+-+
`Cl
`field name
`I
`field Ian
`I
`-+-+-+-+-+-+-+-+-+-+-+-+-+—+-+-+-+-+-+-+-+-+-+-+-+-+-+~+-+-+-+
`field data
`I
`-+-+-+-+-+-+-+-+-+-+-+-+-+—+-+-+-+-+-+-+-+-+-+-+-+-+—+—+-+-+-+
`
`l
`fieid dataf 14“
`-+-+-+-+-+-+-+-+-+-+-+—+-+-+-+-+-+—+-+-+-+-+-+-+-+-+-+-+-+-+-+
`
`FIG. 14A
`
`Page 00016
`
`Page 00016
`
`

`

`US. Patent
`
`Jan. 26, 2010
`
`Sheet 15 of 15
`
`US 7,653,685 B2
`
`1450
`
`/’
`
`1456
`
`1458
`
`1550
`' 3
`
`0
`
`1
`
`2
`
`1454 012&56789gf2345678fi12345678901
`
`-+-+-+-+-+-+-+-+-+-+-+
`w~+-+-+ +-+-+-+-+-+- +-+-+-+-+-+-+-+-+-
`lctfi id ID
`{field [en
`1
`field data
`|
`+~+-+-+-+-+-+-+-+-+-+-+-+- -+-+-+-+-+-+-+-+-+-+-+-
`
`+-+-+-+-+-Ail-+4-+-+-+-+-+-+-+-+-+-+~+-+-+-+-+-+-+-+—+-+-+-+
`lCIfield ID
`{field len
`|
`field data
`I
`I
`...
`l
`
`-+-+-+-+-+—+
`
`FIG. 14B
`
`
`
`+-+-+-+-+-+~+-+-+-+-+-+-+-+-+-+-+-+—+-+-+-+-+-+-+-+-+-+-+-+-+-+
`
`+—+-+-+-+-+Ii-l-+-+-+-+-+-+-+-+—+-+-+1-i§§-+-+-+-+-+-+-+-+-+-+-+
`1401 { lcl
`fiefd name
`i
`field Jen
`I
`1409 { I
`field data
`+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+—+-+-+-+-+-+-+-+-+-+-+-+
`I
`...
`I
`+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
`lleieldlD lfieldien
`l
`field data
`I
`+«+-+—+-+-+-+-+-+-+-+-+-+-+-+-+-+-+~+-+-+-+-+-+-+-+-+-+-+-+-+-+
`‘
`...
`1
`
`I
`
`FIG. 140
`
`Page 00017
`
`Page 00017
`
`

`

`US 7,653,685 B2
`
`1
`METHODS AND APPARATUSES FOR
`TRANSFERRING STREAMING
`MULTIMEDIA DATA AND VARIOUS TYPES
`OF DATA ASSOCIATED THEREWITH IN A
`NETWORK ENVIRONMENT
`
`This application is a divisional application of US. patent
`application Ser. No. 09/757,235, filed on Jan. 8, 2001 now
`US. Pat No. 7,007,062, which is a continuation-in-part of
`US. patent application Ser. No. 09/603,108, filed on Jun. 22,
`2000 now US. Pat No. 7,191,242.
`
`FIELD OF THE INVENTION
`
`The present invention relates to the field ofmultimedia data
`transmission. In particular, the present invention in one exem-
`plary embodiment relates to multimedia data transmission of
`real-time transfer protocol (RTP) packets using real time
`streaming protocol (RTSP) in a computer network environ-
`ment.
`
`INTRODUCTION AND BACKGROUND OF THE
`INVENTION
`
`Methods of transmitting data are commonly known and
`performed today on a routine basis to send various multime-
`dia data such as text, graphics, audio, video, images etc.
`across computer networks situated in various parts of the
`world. Generally the transmission process requires both hard-
`ware and software for performing its function. Typically, the
`hardware includes various types of personal computers and
`hand held multimedia data sending or receiving devices.
`These devices run under the control of an operating system
`and utilize multimedia application software programs. As is
`known in the art, streaming media data is data which is trans-
`mitted to a receiving computer system and presented (usually
`after buffering temporarily at the receiving system) and then
`discarded (not stored) at the receiving system.
`Currently, data is sent in form of packets from one multi-
`media device to another. A large amount of information is
`required to be sent in a real-time system and presented (usu-
`ally after buffering temporarily at the receiving system) and
`then discarded (not stored) at the receiving system.
`Currently, data is sent in form of packets from one multi-
`media device to another. A large amount of information is
`required to be sent in a real-time manner in the data packets,
`which imposes a heavy load on the systems. Streaming media
`data, such as Real-Audio data in streaming media format
`specified by Real-Networks, is sent through the Internet is
`near real-time manner in many cases.
`In one approach, the components involved in data trans-
`mission of streaming media are known to be a server (which
`may be referred to as originating server), a caching proxy
`server and a client. These components in various combina-
`tions communicate with each other for transmitting data
`packets in real-time. The communication link that currently
`exists between the components uses real-time transfer proto-
`cols (RTP) and real-time streaming protocols (RTSP) to com-
`municate and send packets to each other. For this approach to
`work, a caching proxy server needs to communicate with the
`system server, receive a stream of RTP data packets, and
`transfer the information contained within the RTP data pack-
`ets to a client. FIG. 1a shows an example of a prior method in
`which a caching proxy server receives streaming media data
`and provides this data to a client. In order to perform its
`function properly and efficiently, the caching proxy server
`
`2
`
`needs several pieces of information from the server to be able
`to cache an RTP stream easily and reliably.
`A problem with the current approach is that it is not able to
`provide some of the key required information such as data
`packet transmit time and video packet frame type information
`that a caching proxy needs to be efficient. This information
`allows a caching proxy server to provide smooth packet deliv-
`ery to its client by knowing the time an RTP data packet was
`intended to be sent, and type of video frame that is being sent
`without knowing the specific payload format. Another prob-
`lem with the current approach is that it is not able to provide
`multiple pieces of unrelated data in one delivery to the cach-
`ing proxy server. Furthermore, packets from the server may
`be “lost” and never reach the caching proxy server. In addi-
`tion, there is normally no way to recreate a complete “pris-
`tine” copy at the caching proxy server.
`Prior art servers communicate RTP information to the
`
`caching proxy server by sending information through a
`cache-control header.
`In one approach, a cache-control
`header contains normal header fields. In another approach,
`unrelated to cache control of RTP information, a single type
`of additional information has been added to the normal fields
`
`10
`
`15
`
`20
`
`in a header extension format without specifying the type of
`additional information. In this approach only a single piece of
`RTP extension can be added to the normal field of the header
`
`25
`
`and sent at any one time.
`limited, non-extensible
`A problem with using this
`approach is that a server is not able to attach multiple sets of
`unrelated data at a time to send to the caching proxy server.
`Another problem with this approach is that the header exten-
`sion used in these methods are still not able to provide all the
`information a caching proxy server needs to cache a stream
`properly and to transmit the stream properly. Yet another
`problem with this approach is that there is no way to identify
`the particular extension independently of other possible
`extensions.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides several methods and appa-
`ratuses for transmitting multimedia data using streaming
`media protocols such as real-time transfer protocols (RTP)
`and real-time streaming protocols (RTSP) in a computer net-
`work environment. In one exemplary embodiment, a request
`for RTP data is sent from the caching proxy server to the
`server. The request may be for one specific type of data or
`multiple unrelated types of data. The server responds to the
`request indicating its support for the requested RTP data. The
`caching proxy server determines whether to proceed or ter-
`minate the data transmission process based on the response
`provided by the server. If it is determined to proceed with the
`data transmission process, the caching proxy informs the
`server to send the requested and supported RTP data. The
`server sends the requested data in the body of a packet. The
`RTP packet uses a RTP Meta-Info payload format, which
`includes a body and a field header. The field header includes
`fields to identify the streaming media data, and the field body
`includes the requested streaming media data.
`In another embodiment, the caching proxy server requests
`and receives packet transmit time data and/or packet frame
`type data from the server. The caching proxy server uses the
`frame type data to communicate with the client and supply
`frames based on client’s capacity to handle loads at given
`times. Transmit time data is also used by the caching proxy to
`store packets locally and deliver these packets at appropriate
`times to the client for a smooth packet delivery.
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Page 00018
`
`Page 00018
`
`

`

`US 7,653,685 B2
`
`3
`Other features and advantages ofthe present invention will
`be apparent from the accompanying drawings, and from the
`detailed description, which follows below.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention is illustrated by way of example and
`not limited by the figures of the accompanying drawings in
`which like references indicate similar elements and in which:
`
`FIG. 1a is a flowchart which shows a method in the prior art
`for transferring streaming media data to caching proxy server
`and then to a client;
`FIG. 1b illustrates a network of computer systems in which
`media data may be exchanged and/or processed, according to
`one embodiment of the present invention;
`FIG. 2 illustrates a block diagram of an exemplary digital
`processing system, which may be used in accordance with
`one embodiment of the present invention;
`FIG. 3 illustrates one embodiment of a communication
`
`method between a server and a client using RTSP and RTP
`protocols;
`FIG. 4 illustrates another embodiment of a communication
`
`method between a server, caching proxy server and a client;
`FIG. 5 illustrates one embodiment of a RTSP, RTP nego-
`tiation process between a caching proxy and a server;
`FIG. 6 illustrates one embodiment of a relationship
`between the server, caching proxy, and client during a transfer
`of a Transmit Time (TT) sub-extension to the caching proxy
`server and its use ofTT information in transmitting streaming
`data to a client;
`FIG. 7 illustrates one embodiment of process that takes
`place during transfer of a transmit
`time sub-extension
`between server and caching proxy server;
`FIG. 8 illustrates one embodiment of process that takes
`place during transfer of a frame type sub-extension between
`server, and caching proxy server;
`FIG. 9 is a flow diagram of one embodiment ofan operation
`to provide various types of information to a caching proxy in
`an extensible header format;
`FIG. 10 illustrates one embodiment of a relationship
`between the server, caching proxy, and client during a transfer
`of a Frame Type sub-extension;
`FIG. 11 illustrates a block diagram of a machine readable
`medium which stores executable computer program instruc-
`tion for execution by an exemplary caching proxy server,
`which may be used in accordance with one embodiment ofthe
`present invention;
`FIG. 12 illustrates a block diagram of a machine readable
`medium which stores executable computer program instruc-
`tion for execution by an exemplary originating server
`(server), which may be used in accordance with one embodi-
`ment of the present invention;
`FIG. 13 illustrates a block diagram of a machine readable
`medium which stores executable computer program instruc-
`tion for execution by an exemplary client, which may be used
`in accordance with one embodiment of the present invention;
`FIG. 14a illustrates an exemplary RTP data packet includ-
`ing a standard field header and a field body in accordance with
`one embodiment of the present invention;
`FIG. 14b illustrates an exemplary RTP data packet includ-
`ing a compressed field header and a field body in accordance
`with one embodiment of the present invention; and
`FIG. 14c illustrates an exemplary format of a combination
`RTP data packet 1470 in accordance with one embodiment of
`the present invention.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`DETAILED DESCRIPTION
`
`A method and system for providing multimedia data trans-
`mission using real-time transfer protocol (RTP) and real time
`streaming protocol (RTSP) are described. For purposes of
`explanation, numerous specific details are set forth in order to
`provide a thorough understanding of the present invention.
`For example, various computer network system architectures
`and digital processing system architectures are provided for
`illustrative purposes rather than to be construed as limitations
`of the present invention. It will be evident, however, to one
`skilled in the art that the present invention may be practiced
`without these specific details. In other instances, well-known
`structures and devices are shown in block diagram form to
`facilitate explanation.
`FIG. 1b is a diagram of a network of computer systems in
`which media data may be processed, according to one
`embodiment of the present invention. As shown in FIG. 1b, a
`number of client computer system, one or more ofwhich may
`represent one implementation of a receiving system, are
`coupled together through an Internet 122. It will be appreci-
`ated that the term “Internet” refers to a network of networks.
`
`Such networks may use a variety ofprotocols for exchange of
`information, such as TCP/IP, ATM, SNA, SDI, RTP, RTSP
`etc. The physical connections ofthe Internet and the protocols
`and communication procedures of the Internet are well
`known to those in the art. Access to the Internet 103 is typi-
`cally provided by Internet service providers (ISPs), such as
`the ISP 124 and the ISP 126, which may also be connected
`with caching proxy servers 130 and 132. Users on client
`systems, such as the client computer systems 102, 104, 118,
`and 120, generally obtain access to the Internet through Inter-
`net service providers, such as ISPs 124 and 126, which may
`also be connected through the intemet with caching proxy
`servers 130 and 132. Access to the Internet may facilitate
`transfer ofinformation (e. g., email, text files, media files, etc.)
`between two or more digital processing systems, such as the
`client computer systems 102, 104, 118, and 120 and/or a
`streaming media server system 128 which may be considered
`an originating server from which caching proxy servers
`receive streaming media data. For example, one or more of
`the client computer systems 102, 104, 118, and 120 and/or the
`streaming media server 128 may provide media data (e.g.,
`video and audio, or video, or audio) to another one or more of
`the client computer systems 102, 104, 118, and 120 and/or the
`streaming media server 128. Such may be provided in
`response to a request. As described herein, such media data
`may be transferred in the system 100 according tracks. Such
`tracks, in one embodiment of the invention, may be created
`according to a specific format of the streaming media data
`and/or a specific data communication (e. g., network) protocol
`(s).
`The streaming media server 128 is typically comprised of
`at least one computer system to operate with one or more data
`communication protocols, such as the protocols of the World
`Wide Web, and as such, is typically coupled to the Internet
`122. Optionally, the streaming media server 128 may be part
`of an ISP which may provide access to the Internet and/or
`other network for client computer systems. The client com-
`puter systems 102, 104, 118, and 120 may each, with appro-
`priate web browsing software, access data, such as HTML
`documents (e.g., Web pages), which may be provided by the
`streaming media server 128. Such data may provide media,
`such as QuickTime® movies or QuickTime® streaming
`media data, which may be presented by the client computer
`systems 102, 104, 118, and 120.
`
`Page 00019
`
`Page 00019
`
`

`

`US 7,653,685 B2
`
`5
`The ISP 124 provides Internet connectivity to the client
`computer system 102 Via a modem interface 106, which may
`be considered as part of the client computer system 102. The
`client computer system may be a conventional computer sys-
`tem, such as a Macintosh computer, a “networ ” computer, a
`handheld/portable computer, a Web TV system, or other types
`of digital processing systems (e.g., a cellular telephone hav-
`ing digital processing capabilities). Similarly, the ISP 126
`provides Internet connectivity for the client computer sys-
`tems 104, 118 and 120, although as depicted in FIG. 1b, such
`connectivity may vary between various client computer sys-
`tems, such as the client computer systems 102, 104, 118, and
`120. For example, as shown in FIG. 1b, the client computer
`system 104 is coupled to the ISP 126 through a modem
`interface 108, while the client computer systems 118 and 120
`are part of a Local Area Network (LAN). The interfaces 106
`and 108, shown as modems 106 and 108, respectively, in FIG.
`1b, may be an analog modem, an ISDN modem, a cable
`modem, a satellite transmission interface (e.g., “Direct PC”),
`a wireless interface, or other interface for coupling a digital
`processing system, such as a client computer system, to
`another digital processing system. The client computer sys-
`tems 118 and 120 are coupled to a LAN bus 112 through
`network interfaces 114 and 116, respectively. The network
`interfaces 114 and 116 may be an Ethemet-type, Asynchro-
`nous Transfer Mode (ATM), or other type of network inter-
`face. The LAN bus is also coupled to a gateway digital pro-
`cessing system 110, which may provide firewall and other
`Internet-related services for a LAN. The gateway digital pro-
`cessing system 110, in turn, is coupled to the ISP 126 to
`provide Internet connectivity to the client computer systems
`118 and 120. The gateway digital processing system 110 may,
`for example, include a conventional server computer system.
`Similarly, the streaming media server 128 may, for example,
`include a conventional server computer system.
`The system 100 may allow one or more of the client com-
`puter systems 102, 104, 118, and 120 and/or the streaming
`media server 128 to provide media data (e.g., video and audio,
`or video, or audio) to another one or more of the client
`computer systems 102, 104, 118, and 120 and/or the stream-
`ing media server 128. Such data may be provided, for
`example, in response to a request by a receiving system,
`which may be, for example, one or more of the client com-
`puter systems 102, 104, 118, and 120.
`FIG. 2 is a block diagram of an exemplary digital process-
`ing system which may be used in accordance with one
`embodiment ofthe present invention. For example, the digital
`processing system 250 shown in FIG. 2 may be used as a
`client computer system, a streaming media server system, a
`conventional server system, etc. Furthermore, the digital pro-
`cessing system 250 may be used to perform one or more
`functions of an Internet service provider, such as the ISP 124
`or 126. The digital processing system 250 may be interfaced
`to external systems through a modem or network interface
`268. It will be appreciated that the modem or network inter-
`face 268 may be considered as part of the digital processing
`system 250. The modem or network interface 168 may be an
`analog modem, an ISDN modem, a cable modem, a token
`ring interface, a satellite transmission interface, a wireless
`interface, or other interface(s) for providing a data commu-
`nication link between two or more digital processing systems.
`The digital processing system 250 includes a processor
`252, which may represent one or more processors and may
`include one or more conventional types of such processors,
`such as a Motorola PowerPC processor, an Intel Pentium (or
`x86) processor, etc. A memory 255 is coupled to the processor
`252 by a bus 256. The memory 255 may be a dynamic random
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`access memory (DRAM) and/or may include static RAM
`(SRAM). The processor may also be coupled to other types of
`storage areas/memories (e. g., cache, Flash memory, disk,
`etc.), which could be considered as part ofthe memory 255 or
`sep