United States Patent
`(12)
`(10) Patent No.:
`US 7,191,242 B1
`
`Serenyi et al.
`(45) Date of Patent:
`Mar. 13, 2007
`
`US007191242B1
`
`(54) METHODS AND APPARATUSES FOR
`TRANSFERRING DATA
`
`(75)
`
`Inventors: Denis Serenyi, Menlo Park, CA (US);
`Chris LeCroy, Boulder Creek, CA
`(US)
`
`.
`.
`(73) A551gnee: Apple, Inc., Cupertmo, CA (US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 716 days.
`
`(21) Appl. No.: 09/603,108
`
`(22)
`
`Filed:
`
`Jun_ 22, 2000
`
`(51)
`
`Int. Cl.
`(2006.01)
`G06F 15/16
`(52) US. Cl.
`....................................... 709/231; 709/217
`(58) Field of Classification Search ................ 709/223,
`.
`.
`70933132245225
`See appllcat1on file for complete search h1story~
`-
`References Cited
`
`(56)
`
`US. PATENT DOCUMENTS
`709/228
`5 918 020 A *
`6/1999 Blackard et al
`6,252,889 B1*
`6/2001 Patki et 31.
`...'....W..... 370/474
`
`7/2001 Geagan et 31. ........... 709/231
`6,263,371 B1 *
`
`..... 370/248
`6,275,471 B1*
`8/2001 Bushmitch et al.
`.
`6,300,887 B1* 10/2001 Le ........................... 341/60
`
`
`3/2002 Huckins .................. 348/512
`6,359,656 B1 *
`~~~~~~~~ 709/231
`6,415,327 131*
`7/2002 Beekerman et 31~
`6,505,169 Bl *
`1/2003 BhagaVath et 3T
`6’546’421 131*
`£2003 Wynblattlet 31' """""" 70/9/225
`
`“ 37790382;
`22333: E; *
`35883 $631561 a1””””
`6,639,896 B1* 10/2003 Goode et al.
`............... 370/224
`6,674,477 B1 *
`1/2004 Yamaguchi et al.
`..... 348/3871
`6,708,213 B1*
`3/2004 Bommajah et a1.
`......... 709/226
`
`3/2004 Shiobara ..................... 370/235
`6,711,131 B1*
`8/2004 Brassil et al.
`............... 370/390
`6,771,644 B1 *
`FOREIGN PATENT DOCUMENTS
`
`W0
`W0
`
`6/1997
`WO 97/22201 A2
`5 2000
`W0 00 27087 A1
`/
`/
`OTHER PUBLICATIONS
`
`Schulzrinne et 31., “RTP: A Transport Protocol for Real-Time
`
`APPIlcatlonS”a Jan' 1396’ RFC 1889*
`$533110? lgggliRFET;5131:1026 Format for MPEGl/MPEGZ
`’
`'
`’
`'
`(Continued)
`
`Primary Examinerilason Cardone
`Assistant ExamineriThomas Duong
`y g
`y
`y
`74 Allorne ,A 6111, or FirmiBlakel , Sokoloif, Ta lor &
`Zafman LLP
`
`ABSTRACT
`(57)
`The present invention provides several methods and appa-
`ratuses for transmitting multimedia data using streaming
`media protocols such as real-time transfer protocols (RTP)
`and real-time streamin
`rotocols RTSP in a com uter
`.
`g p
`.
`p
`network enV1ronment. In one exemplary embodlment, a
`request for RTP data and its associated extension is sent from
`the caching proxy server to the server. The request may be
`for one Spedfic type 0f data or muniple unrelated types 0f
`data. The server responds to the request
`indicating its
`SHPPOrt for the requested RTP extension data- The caching
`proxy server determines whether to proceed or terminate the
`data transmission process based on the response provided by
`the server. If it
`is determined to proceed with the data
`transmission process, the caching proxy informs the server
`to send the requested and supported RTP data. The server
`sends the requested data in a variable and extendible header
`format
`
`42 Claims, 13 Drawing Sheets
`
`CF ASKS SETUP IN RTSP
`FOR AUDIO DATA AND SPECIFIES
`EXTENSIONS TO RTP DATA
`FOR AUDIO DATA
`
`501
`
`502
`
`503
`
`502
`
`504
`
`Apple Exhibit 4220
`
`Apple V. SightSound Technologies
`CBM2013-00020
`
`Page 00001
`
`EXTENSIONS 505
`
`SERVER RESPONDS/NEGOTIATES
`
`GP ASKS SETUP IN RTSP
`FOR VIDEO DATA AND SPECIFIES
`EXTENSIONS T0 RTP DATA
`
`SERVER RESPONDS/NEGOTIATES
`
`CP ASKS PLAY IN RTSP
`
`SERVER RESPONDS WITH RTP DATA
`FOR ALL TRACKS WITH NEGOTIATED
`
`Apple Exhibit 4220
`Apple v. SightSound Technologies
`CBM2013-00020
`Page 00001
`
`

`

`US 7,191,242 B1
`Page 2
`
`OTHER PUBLICATIONS
`
`Schulzrinne, H. et al, “RTP: A Transport Protocol for Real-Time
`Applications”, Jan. 1 996, RFC 1889*
`Carsten Griwodz, Alex Jonas, Michael Zink, Affordable Infrastruc-
`ture for Stream Playback in the Internet, Darmstadt University of
`Industrial Process and System Communications (KOM), Dec. 12,
`1999, Technical Report TR-KOM-l999-07, Darmstadt, Germany.
`Schulzrinne, Henning and Rosenberg, Jonathan. “Internet Tele-
`phony: architecture and protocolsian IETF perspective,” Com—
`
`purer Networks, ElseVier Science Publishers: Amsterdam, NL, V01.
`31 No. 3, Feb. 11, 1999 (pp. 237—255).
`.
`.
`.
`“
`.
`Schulzrinne, Henning et al. RTP. Atransport protocol for real-time
`applications,” Network Working Group RFC 1889 RTP’ Jan. 1996
`(pp. 1'75)
`PCT International Search Report for PCT Int’l Appln No. US01/
`20044, mailed May 22, 2002 (8 pages).
`
`* Cited by examiner
`
`Page 00002
`
`Page 00002
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 1 of 13
`
`US 7,191,242 B1
`
`ORIGINATING SERVER AND CACHING PROXY SERVER(CP)
`NEGOTIATE, THROUGH REAL-TIME STREAMING PROTOCOL
`(RTSP) FOR SERVER TO SEND AND OF SERVER TO RECEIVE
`STREAMING MEDIA DATA (FOR‘EXAMPLE: OUICKTIME
`STREAMING AUDIO DATA).
`
`STREAMING MEDIA DATA TO THE CLIENT SYSTEM.
`
`CP SERVER RECEIVES STREAMING MEDIA DATA, USING REAL
`TIME TRANSFER PROTOCOL (RTP), IN STREAMING MEDIA
`FORMAT NOT THE SAME FORMAT AS STORED ON ORIGINATING
`
`SERVER AND CP SERVER STORES THE STREAMING MEDIA DATA,
`IN THE STREAMING MEDIA FORMAT, ON A STORAGE DEVICE
`CONTROLLED BY CP SERVER.
`
`CP SERVER RECEIVES REQUEST, THROUGH RTSP, FROM A
`CLIENT FOR THE STREAMING MEDIA DATA AND RETRIEVES THE
`
`DATA FROM THE STORAGE DEVICE AND TRANSMITS THE
`
`FIG. 1A
`
`Page 00003
`
`Page 00003
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 2 of 13
`
`US 7,191,242 B1
`
`100
`
`\
`
`122
`
`
`
`
`
`122
`
`130
`(\J
`CP SERVER
`
`
`
`
`102
`
`106
`
`CLIENT
`COMPUTER W
`SYSTEM
`
`‘04
`
`108
`
`CLIENT
`SYSTEM W
`COMPUTER
`
`110
`
`GATEWAY
`SYSTEM
`
`1 12
`
`INTERNET
`
`132
`
`(\J
`0P SERVER
`-
`
`126
`
`128
`
`
`STREAMING
`
`MEDIA SERVER
`SYSTEM
`
`114
`
`116
`
`NETWORK
`INTERFACE
`
`NETWORK
`INTERFACE
`
`118
`
`120
`
`CLIENT
`COMPUTER
`SYSTEM
`
`CLIENT
`COMPUTER
`SYSTEM
`
`Page 00004
`
`Page 00004
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 3 of 13
`
`US 7,191,242 B1
`
` 255
` 252
`
`
`
`. 26
`
`l
`
`258
`
`MEMORY
`
`262
`
`4
`
` PROCESSOR
`
`
`DISPLAY
`CONTROLLER
`
`DISPLAY
`
`MASS
`MEMORY
`
`|/O
`CONTROLLER
`
`266
`
`VD
`DEVICES
`
`260
`
`
`
`
`
`
` DIGITAL PROCESSING
`SYSTEM 259
`
` MODEM OR
`
`NETWORK
`
`
`INTERFACE
`
`
`FIG. 2
`
`Page 00005
`
`Page 00005
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 4 of 13
`
`US 7,191,242 B1
`
`301
`
`302
`
`RTSP & RTP
`
`
`
`SERVER(S)
`
`CLIENT(C)
`
`FIG. 3
`
`301
`
`401
`
`302
`
`CLIENT(C)
`
`CACHING PROXY
`SERVER (GP)
`
`SERVER(S)
`
`RTSP & RTP
`
`FIG. 4
`
`Page 00006
`
`Page 00006
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 5 of 13
`
`US 7,191,242 B1
`
`CP ASKS SETUP IN RTSP
`
`FOR AUDIO DATA AND SPECIFIES
`EXTENSIONS TO RTP DATA
`
`
`
`FOR AUDIO DATA
`
`SERVER RESPONDS/NEGOTIATES
`
`
`
`
`CP ASKS SETUP IN RTSP
`FOR VIDEO DATA AND SPECIFIES
`EXTENSIONS TO RTP DATA
`
`SERVER RESPONDS/NEGOTIATES
`
`CP ASKS PLAY IN RTSP
`
`SERVER RESPONDS WITH RTP DATA
`
`EXTENSIONS
`
`FOR ALL TRACKS WITH NEGOTIATED
`
`501
`
`502
`
`503
`
`502
`
`504
`
`505
`
`FIG. 5
`
`Page 00007
`
`Page 00007
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet6 0f13
`
`US 7,191,242 B1
`
`Emfio
`
`mEE
`
`mom5vSm
` mm>mmw
`
`Hmkuqm
`
`h2mé$fi
`
`mom
`
`
`
`@5on.203
`
`noEwmem
`
`
`
`
`
`
`
`52me$5meSEER_>_<mE.m
`
`
`
`t.._.<.2630
`
`
`
`o._.<H<o
`
`@252me0mmx0mmmHEP0mmowow
`02%Ct.Ctxmro453<H<o<H<o«EB
`
`
`
`Page 00008
`
`Page 00008
`
`
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 7 of 13
`
`US 7,191,242 B1
`
`
`
`CP ASKS FOR TRANSMIT TIME
`INFORMATION FROM THE SERVER
`
`701
`
`
`
`
`
`
`
`
`
`
`
`
`
`705
`
`706
`
`____________ -*—I
`I
`I
`CLIENT MAY ALSO
`:
`:
`ASK FOR PLAY OR
`I
`w. OTHER DATA DIRECTLY I
`I
`PROM SERVER
`I
`711
`_____________f_.I
`
`
`702.—___.—___________—____—_—___—_.
`
`SERVER RECEIVES THE REQUEST FOR
`
`TRANSMIT TIME (TI)
`709
`
`
`SERVER DOES NOT RESPOND TO
`
`
`SERVER RESPONDS AND SENDS TT
`REQUEST THIS INDICATES THAT
`
`
`IDENTIFIER CODE, WHICH
`SERVER DOES NOT SUPPORT THE
`
`
`
`CORRESPONDS, TO TTTYPE OF
`EXTENSION REQUESTED
`
`
`
`EXTENSION, IN AN EXTENSIBLE
`
`
`HEADER FORMAT - STREAMING DATA
`
`
`IS SENT WITH TT EXTENDED DATA
`CP MAY TERMINATE PROCESS
`
`
`710
`IF "IT IS CRITICAL ELEMENT
`
`
`OF THE CP OPERATION
` -EXTENSIBLE HEADER FORMAT IS A
`
`
`SUB—EXTENSION CONTAINING A SUB-
`
`EXTENSION ID, IDENTIFYING THE
`
`CONTENT OF THE INFORMATION.
`—I'I' SUB-EXTENSION ALSO
`CONTAINS A SINGLE 64-BIT
`UNSIGNED INTEGER REPRESENTING
`RECOMMENDED TRANSMISSION
`TIME OF RTP PACKET IN
`MILLISECONDS
`
`
`
`
`CP GETS THETTDATA, REMDVES
`
`
`EXTENSION HEADER AND STORES THE
`
`
`DATA LOCALLY;TITIMES ARE
`ASSOCIATED WITH APPROPRIATE
`
`STREAMING DATA
`I
`
`
`
`CLIENT REQUESTS STREAMING DATA
`BY MAKING A PLAY REQUEST IN
`
`RTSP
`
`CP RESPONDS AND SENDS TRACKS OF DATA AT APPROPRIATE TIMES
`SPECIFIED BY TTTIMES ASSOCIATED WITH STREAMING DATA
`
`III I
`
`;
`
`. I
`
`707
`
`CLIENT PLAYS DATA
`
`I
`708
`—————————————————————————————
`
`Page 00009
`
`Page 00009
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 8 of 13
`
`US 7,191,242 B1
`
`
`
`CP ASKS FOR FRAME TYPE
`INFORMATION FROM THE SERVER
`
`
`
`SERVER RECEIVES THE REQUEST FOR —————————————————————————————
`FRAME TYPE (FT)
`
`807
`
`
`SERVER DOES NOT RESPOND TO
`
`
`SERVER RESPONDS AND SENDS FT
`REQUEST THIS INDICATES THAT
`
`IDENTIFIER CODE. WHICH
`SERVER DOES NOT SUPPORT THE
`
`
`
`CORRESPONDS, TO FT TYPE OF
`EXTENSION REOUESTED
`
`
`
`EXTENSION, IN AN EXTENSIBLE
`HEADER FORMAT - STREAMING DATA
`
`IS SENT WITH FT EXTENDED DATA
`OF MAY TERMINATE PROCESS
`
`IF FT IS CRITICAL ELEMENT
`
`
`
`EXTENSIBLE HEADER FORMAT IS A
`OF THE OF OPERATION
`
`
`SUB-EXTENSION CONTAINING A SUB-
`EXTENSION ID, IDENTIFYING THE
`
`
`CONTENT OF THE INFORMATION.
`FT SUB-EXTENSION ALSO
`
`CONTAINS 64-BIT UNSIGNED INTEGER
`
`
`VALUE WITH WELL-KNOWN FRAME
`TYPES SUCH AS O-UNKNOWN FRAME,
`1-KEY FRAME, 2-B FRAME, AND
`
`
`
`3-P FRAME. FUTURE VERSIONS
`CLIENT INFORMS 45,09
`CAN ADD MORE FRAMES.
`
`OF THAT IT IS
`
`OVERLOADEO
`
`r ------------'—
`WITH
`
`
`.
`CLIENT MAY ALSO
`INFORMATION
`
`ASK FORFTOR
`
`I
`CP GETS THE FT DATA REMOVES
`
`EXTENSION HEADER AND STORES THE
`N. OTHER DATA DIRECTLY
`
`810 I
`FROMTHESERVER
`DATA LOCALLY; FT FRAMES ARE
`ASSOCIATED WITH APPROPRIATE
`STREAMING DATA
`
`
`
`
` CP REVIEWS CLIENT REQUEST STOPS
`SENDING LEAST IMPORTANT FRAMES
`
`AND SENDS ONLY HIGHER
`IMPORTANCE FRAMES
`
`----------------------------
`
`II
`
`I|
`
`Il
`
`808
`
`
`
`
`
`
`FIG. 8
`
`Page 00010
`
`Page 00010
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 9 of 13
`
`US 7,191,242 B1
`
`CP ASKS BY NAME FOR ONE OR MORE RTP
`SUB-EXTENSIONS USING RTSP PROTOCOL
`
`901
`
`
`
`SERVER RESPONDS BACK TO CP INDICATING ITS
` 903
`SUPPORT FOR THE REQUESTED SUB-EXTENSIONS.
`
`
`SERVER ALSO TRANSMITS T0 OF AN IDENTIFIER (E.G.,
`
`NUMBER CODE) CORRESPONDING TO EACH NAME
`
`OF RTP EXTENSION. CP MAY LATER USE THE CODE
`WHEN RECEIVING EXTENDED DATA. RESPONSE
`MAY OR MAY NOT BE A COMPLETE RESPONSE.
`
`
`
`
`
`
`
`I
`CP RECEIVES SERVER’S RESPONSE
`
`905
`
`
`
`907
`
`
`
`
`
`
`
`
`
`
`
`GP EVALUATES CLIENT REQUESTS AND
`PERFORMS FUNCTIONS ACCORDINGLY.
`FOR EXAMPLE FRAME THINNING OR USING
`929
`TRANSMIT TIME OPTIONALLY TO TRANSMIT
`
`DATA
`
`
`
`931
`
`FIG- 9
`
`Page 00011
`
`CP DETERMINES WHETHER SERVER RESPONDED
`909
`TO ALL REQUESTED SUB-EXTENSIONS
`
`
`
`NO
`
`YES
`
`
`SERVER DID NOT RESPOND TO ALL
`SERVER RESPONDED TO ALL
`
`
`
`
`REQUESTED SUB-EXTENSIONS
`REQUESTED SUB-EXTENSIONS
`
`
`I
`
`9 1
`CP REQUESTS SERVER TO SEND ALL
`2
`ARE ANY OF THE MISSING SUB-
`
`
`SUPPORTED STREAMING MEDIA DATA
`EXTENSIONS CRITICAL TO CP
`
`
`ASSOCIATED THE RTP SUB-EXTENSIONS
`PROCESSES?
`
`
`
`
`
`
`SERVER SENDS ALL SUPPORTED RTP SUB-
`911
`
`EXTENSIONS AND STREAMING MEDIA DATA
`
`
`ASSOCIATED WITH THE RTP SUB-
`
`EXTENSIONS
`913
`
`
`
`
`CP
`
`CP STORES SUB-EXTENSION DATA AND
`
`CP DECIDES TO PROCEED
`TERMINATES
`STREAMING MEDIA ASSOCIATED WITH THE
`AND REQUEST THE
`
`
`RTP SUB-EXTENSION LOCALLY
`OPERATIONS
`REMAINING/SUPPORTED
`
`
`SUB-EXTENSIONS
`927
`I
`
`OF MAY REMOVE SUB-EXTENSION FIELDS
`917
`
`
`
`
`CP DECIDES WHETHER
`TO TERMINATE OPERATIONS
`
`915
`
`Page 00011
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 10 of 13
`
`US 7,191,242 B1
`
`mom
`
`OhnommrEozm§$t
`
`
`
`$5252::“#5:
`
`
`
`.2630.EvGE
`
`
`
` #530$555
`
`ESQ/22:
`
`mum
`
`mum
`
`m=>_<mn_
`
`m=2<E
`
`m_>_<mn_
`
`$an
`
`WES”:
`
`:9528525mwEx>20
`
`8szmo
`
`2..2".
`
`
`
`mEon.203
`
`noopmmmEb
`
`Page 00012
`
`Page 00012
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 11 0f 13
`
`US 7,191,242 B1
`
`401
`
`CACHING PROXY SERVER (CF)
`
`DATA 2
`
`TRANSMISSION/RECEIPT
`CONTROL SOFTWARE
`(E.G. FOR RTSP AND RTP)
`
`STREAMING MEDIA
`DATA 2
`
`(IN STREAMING
`MEDIA FORMAT)
`
`TT ASSOCIATED WITH
`STREAMING MEDIA
`DATA 2
`
`FT ASSOCIATED WITH
`STREAMING MEDIA
`
`STREAMING MEDIA
`DATA 1
`
`(IN STREAMING
`MEDIA FORMAT)
`
`1T ASSOCIATED WITH
`STREAMING MEDIA
`DATA 1
`
`FT ASSOCIATED WITH
`STREAMING MEDIA
`DATA 1
`
`FIG. 11
`
`Page 00013
`
`Page 00013
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 12 0f 13
`
`US 7,191,242 B1
`
`301
`
`SERVER (ORIGINATING SERVER)
`
`TRANSCEIVER
`CONTROL SOFTWARE
`
`EXTENSIONS
`
`ORIGINAL DATA FILE
`FOR MEDIA
`
`(IN ORIGINAL
`DATA FORMAT)
`
`RTP EXTENSIONS
`
`NAME
`TT
`FT
`
`CODE
`01
`02
`
`STREAMING MEDIA
`CONVERSION
`
`SOFTWARE (CONVERTS
`ORIGINAL FILE TO
`STREAMING FORMAT)
`
`SOFTWARE FOR
`TRANSMISSION AND
`CONTROL OF RTP
`
`FIG. 12
`
`Page 00014
`
`Page 00014
`
`

`

`U.S. Patent
`
`Mar. 13, 2007
`
`Sheet 13 of 13
`
`US 7,191,242 B1
`
`302
`
`SOFFWARE
`
`DATA PLAYER
`
`SOFTWARE
`
`CLIENT
`
`-/\/
`
`STREAMING MEDIA
`
`WEB BROWSER
`
`1303
`
`FIG. 13
`
`Page 00015
`
`Page 00015
`
`

`

`US 7,191,242 B1
`
`1
`METHODS AND APPARATUSES FOR
`TRANSFERRING DATA
`
`FIELD OF THE INVENTION
`
`The present invention relates to the field of multimedia
`data transmission. In particular, the present invention in one
`exemplary embodiment relates to multimedia data transmis-
`sion of real-time transfer protocol (RTP) packets using real
`time streaming protocol (RTSP) in a computer network
`environment.
`
`INTRODUCTION AND BACKGROUND OF THE
`INVENTION
`
`Methods of transmitting data are commonly known and
`performed today on a routine basis to send various multi-
`media data such as text, graphics, audio, video, images etc.
`across computer networks situated in various parts of the
`world. Generally the transmission process requires both
`hardware and software for performing its function. Typi-
`cally, the hardware includes various types of personal com-
`puters and hand held multimedia data sending or receiving
`devices. These devices run under the control of an operating
`system and utilize multimedia application software pro-
`grams. As is known in the art, streaming media data is data
`which is transmitted to a receiving computer system and
`presented (usually after buffering temporarily at the receiv-
`ing system) and then discarded (not stored) at the receiving
`system.
`in form of packets from one
`Currently, data is sent
`multimedia device to another. A large amount of information
`is required to be sent in a real-time manner in the data
`packets, which imposes a heavy load on the systems.
`Streaming media data, such as Real-Audio data in streaming
`media format specified by Real-Networks, is sent through
`the Internet is near real-time manner in many cases.
`In one approach, the components involved in data trans-
`mission of streaming media are known to be a server (which
`may be referred to as originating server), a caching proxy
`server and a client. These components in various combina-
`tions communicate with each other for transmitting data
`packets in real-time. The communication link that currently
`exists between the components uses real-time transfer pro-
`tocols (RTP) and real-time streaming protocols (RTSP) to
`communicate and send packets to each other. For this
`approach to work, a caching proxy server needs to commu-
`nicate with the system server, receive a stream of RTP data
`packets, and transfer the information contained within the
`RTP data packets to a client. FIG. 1a shows an example of
`a prior method in which a caching proxy server receives
`streaming media data and provides this data to a client. In
`order to perform its function properly and efficiently, the
`caching proxy server needs several pieces of information
`from the server to be able to cache an RTP stream easily and
`reliably.
`A problem with the current approach is that it is not able
`to provide some of the key required information such as data
`packet transmit time and video packet frame type informa-
`tion that a caching proxy needs to be efficient. This infor-
`mation allows a caching proxy server to provide smooth
`packet delivery to its client by knowing the time an RTP data
`packet was intended to be sent, and type of video frame that
`is being sent without knowing the specific payload format.
`Another problem with the current approach is that it is not
`able to provide multiple pieces of unrelated data in one
`delivery to the caching proxy server. Furthermore, packets
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`from the server may be “lost” and never reach the caching
`proxy server. In addition,
`there is normally no way to
`recreate a complete “pristine” copy at the caching proxy
`server.
`Prior art servers communicate RTP information to the
`
`caching proxy server by sending information through a
`cache-control header.
`In one approach, a cache-control
`header contains normal header fields. In another approach,
`unrelated to cache control of RTP information, a single type
`of additional information has been added to the normal fields
`
`in a header extension format without specifying the type of
`additional information. In this approach only a single piece
`of RTP extension can be added to the normal field of the
`
`header and sent at any one time.
`limited, non-extensible
`A problem with using this
`approach is that a server is not able to attach multiple sets of
`unrelated data at a time to send to the caching proxy server.
`Another problem with this approach is that
`the header
`extension used in these methods are still not able to provide
`all the information a caching proxy server needs to cache a
`stream properly and to transmit the stream properly. Yet
`another problem with this approach is that there is no way
`to identify the particular extension independently of other
`possible extensions.
`SUMMARY OF THE INVENTION
`
`The present invention provides several methods and appa-
`ratuses for transmitting multimedia data using streaming
`media protocols such as real-time transfer protocols (RTP)
`and real-time streaming protocols (RTSP) in a computer
`network environment. In one exemplary embodiment, a
`request for RTP data is sent from the caching proxy server
`to the server. The request may be for one specific type of
`data and its related extensions or multiple unrelated types of
`data and their related extensions. The server responds to the
`request indicating its support for the requested RTP data.
`The caching proxy server determines whether to proceed or
`terminate the data transmission process based on the
`response provided by the server. If it
`is determined to
`proceed with the data transmission process,
`the caching
`proxy informs the server to send the requested and supported
`RTP data. The server sends the requested data in a variable
`and extendible header format.
`
`In another embodiment, the caching proxy server requests
`and receives packet transmit time data and/or packet frame
`type data from the server. The caching proxy server uses the
`frame type data to communicate with the client and supply
`frames based on client’s capacity to handle loads at given
`times. Transmit time data is also used by the caching proxy
`to store packets locally and deliver these packets at appro-
`priate times to the client for a smooth packet delivery.
`Other features and advantages of the present invention
`will be apparent from the accompanying drawings, and from
`the detailed description, which follows below.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention is illustrated by way of example
`and not limited by the figures of the accompanying drawings
`in which like references indicate similar elements and in
`which:
`
`FIG. 1a is a flowchart which shows a method in the prior
`art for transferring streaming media data to caching proxy
`server and then to a client.
`
`FIG. 1b illustrates a network of computer systems in
`which media data may be exchanged and/or processed,
`according to one embodiment of the present invention.
`
`Page 00016
`
`Page 00016
`
`

`

`US 7,191,242 B1
`
`3
`FIG. 2 illustrates a block diagram of an exemplary digital
`processing system, which may be used in accordance with
`one embodiment of the present invention.
`FIG. 3 illustrates one embodiment of a communication
`
`method between a server and a client using RTSP and RTP
`protocols.
`FIG. 4 illustrates another embodiment of a communica-
`
`tion method between a server, caching proxy server and a
`client.
`
`FIG. 5 illustrates one embodiment of a RTSP, RTP nego-
`tiation process between a caching proxy and a server.
`FIG. 6 illustrates one embodiment of a relationship
`between the server, caching proxy, and client during a
`transfer of a Transmit Time (TT) sub-extension to the
`caching proxy server and its use of TT information in
`transmitting streaming data to a client.
`FIG. 7 illustrates one embodiment of process that takes
`place during transfer of a transmit
`time sub-extension
`between server and caching proxy server.
`FIG. 8 illustrates one embodiment of process that takes
`place during transfer of a frame type sub-extension between
`server, and caching proxy server.
`FIG. 9 is a flow diagram of one embodiment of an
`operation to provide various types of information to a
`caching proxy in an extensible header format.
`FIG. 10 illustrates one embodiment of a relationship
`between the server, caching proxy, and client during a
`transfer of a Frame Type sub-extension.
`FIG. 11 illustrates a block diagram of a machine readable
`medium which stores executable computer program instruc-
`tion for execution by an exemplary caching proxy server,
`which may be used in accordance with one embodiment of
`the present invention.
`FIG. 12 illustrates a block diagram of a machine readable
`medium which stores executable computer program instruc-
`tion for execution by an exemplary originating server
`(server), which may be used in accordance with one embodi-
`ment of the present invention.
`FIG. 13 illustrates a block diagram of a machine readable
`medium which stores executable computer program instruc-
`tion for execution by an exemplary client, which may be
`used in accordance with one embodiment of the present
`invention.
`
`DETAILED DESCRIPTION
`
`A method and system for providing multimedia data
`transmission using real-time transfer protocol (RTP) and real
`time streaming protocol (RTSP) are described. For purposes
`of explanation, numerous specific details are set forth in
`order to provide a thorough understanding of the present
`invention. For example, various computer network system
`architectures and digital processing system architectures are
`provided for illustrative purposes rather than to be construed
`as limitations of the present invention. It will be evident,
`however, to one skilled in the art that the present invention
`may be practiced without these specific details. In other
`instances, well-known structures and devices are shown in
`block diagram form to facilitate explanation.
`FIG. 1b is a diagram of a network of computer systems in
`which media data may be processed, according to one
`embodiment of the present invention. As shown in FIG. 1b,
`a number of client computer system, one or more of which
`may represent one implementation of a receiving system, are
`coupled together through an Internet 122. It will be appre-
`ciated that the term “Internet” refers to a network of net-
`
`works. Such networks may use a variety of protocols for
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`exchange of information, such as TCP/IP, ATM, SNA, SDI,
`RTP, RTSP etc. The physical connections of the Internet and
`the protocols and communication procedures of the Internet
`are well known to those in the art. Access to the Internet 103
`
`is typically provided by Internet service providers (ISPs),
`such as the ISP 124 and the ISP 126, which may also be
`connected with caching proxy servers 130 and 132. Users on
`client systems, such as the client computer systems 102, 104,
`118, and 120, generally obtain access to the Internet through
`Internet service providers, such as ISPs 124 and 126, which
`may also be connected through the internet with caching
`proxy servers 130 and 132. Access to the Internet may
`facilitate transfer of information (e.g., email,
`text
`files,
`media files, etc.) between two or more digital processing
`systems, such as the client computer systems 102, 104, 118,
`and 120 and/or a streaming media server system 128 which
`may be considered an originating server from which caching
`proxy servers receive streaming media data. For example,
`one or more of the client computer systems 102, 104, 118,
`and 120 and/or the streaming media server 128 may provide
`media data (e.g., video and audio, or video, or audio) to
`another one or more of the client computer systems 102,
`104, 118, and 120 and/or the streaming media server 128.
`Such may be provided in response to a request. As described
`herein, such media data may be transferred in the system 100
`according tracks. Such tracks, in one embodiment of the
`invention, may be created according to a specific format of
`the streaming media data and/or a specific data communi-
`cation (e.g., network) protocol(s).
`The streaming media server 128 is typically comprised of
`at least one computer system to operate with one or more
`data communication protocols, such as the protocols of the
`World Wide Web, and as such, is typically coupled to the
`Internet 122. Optionally, the streaming media server 128
`may be part of an ISP which may provide access to the
`Internet and/or other network for client computer systems.
`The client computer systems 102, 104, 118, and 120 may
`each, with appropriate web browsing software, access data,
`such as HTML documents (e.g., Web pages), which may be
`provided by the streaming media server 128. Such data may
`provide media, such as QuickTime movies or QuickTime
`streaming media data, which may be presented by the client
`computer systems 102, 104, 118, and 120.
`The ISP 124 provides Internet connectivity to the client
`computer system 102 via a modem interface 106, which may
`be considered as part of the client computer system 102. The
`client computer system may be a conventional computer
`system, such as a Macintosh computer, a “network” com-
`puter, a handheld/portable computer, a Web TV system, or
`other types of digital processing systems (e.g., a cellular
`telephone having digital processing capabilities). Similarly,
`the ISP 126 provides Internet connectivity for the client
`computer systems 104, 118 and 120, although as depicted in
`FIG. 1b, such connectivity may vary between various client
`computer systems, such as the client computer systems 102,
`104, 118, and 120. For example, as shown in FIG. 1b, the
`client computer system 104 is coupled to the ISP 126
`through a modem interface 108, while the client computer
`systems 118 and 120 are part of a Local Area Network
`(LAN). The interfaces 106 and 108, shown as modems 106
`and 108, respectively, in FIG. 1b, may be an analog modem,
`an ISDN modem, a cable modem, a satellite transmission
`interface (e.g., “Direct PC”), a wireless interface, or other
`interface for coupling a digital processing system, such as a
`client computer system, to another digital processing sys-
`tem. The client computer systems 118 and 120 are coupled
`to a LAN bus 112 through network interfaces 114 and 116,
`
`Page 00017
`
`Page 00017
`
`

`

`US 7,191,242 B1
`
`5
`respectively. The network interfaces 114 and 116 may be an
`Ethemet-type, Asynchronous Transfer Mode (ATM), or
`other type of network interface. The LAN bus is also
`coupled to a gateway digital processing system 110, which
`may provide firewall and other Internet-related services for
`a LAN. The gateway digital processing system 110, in turn,
`is coupled to the ISP 126 to provide Internet connectivity to
`the client computer systems 118 and 120. The gateway
`digital processing system 110 may, for example, include a
`conventional server computer system. Similarly, the stream-
`ing media server 128 may, for example, include a conven-
`tional server computer system.
`The system 100 may allow one or more of the client
`computer systems 102, 104, 118, and 120 and/or the stream-
`ing media server 128 to provide media data (e.g., video and
`audio, or video, or audio) to another one or more of the client
`computer systems 102, 104, 118, and 120 and/or the stream-
`ing media server 128. Such data may be provided, for
`example, in response to a request by a receiving system,
`which may be, for example, one or more of the client
`computer systems 102, 104, 118, and 120.
`FIG. 2 is a block diagram of an exemplary digital pro-
`cessing system which may be used in accordance with one
`embodiment of the present
`invention. For example,
`the
`digital processing system 250 shown in FIG. 2 may be used
`as a client computer system, a streaming media server
`system, a conventional server system, etc. Furthermore, the
`digital processing system 250 may be used to perform one
`or more functions of an Internet service provider, such as the
`ISP 124 or 126. The digital processing system 250 may be
`interfaced to external systems through a modem or network
`interface 268. It will be appreciated that the modem or
`network interface 268 may be considered as part of the
`digital processing system 250. The modem or network
`interface 168 may be an analog modem, an ISDN modem,
`a cable modem, a token ring interface, a satellite transmis-
`sion interface, a wireless interface, or other interface(s) for
`providing a data communication link between two or more
`digital processing systems.
`The digital processing system 250 includes a processor
`252, which may represent one or more processors and may
`include one or more conventional types of such processors,
`such as a Motorola PowerPC processor, an Intel Pentium (or
`x86) processor, etc. A memory 255 is coupled to the pro-
`cessor 252 by a bus 256. The memory 255 may be a dynamic
`random access memory (DRAM) and/or may include static
`RAM (SRAM). The processor may also be coupled to other
`types of storage areas/memories (e.g., cache, Flash memory,
`disk, etc.), which could be considered as part of the memory
`255 or separate from the memory 255.
`The bus 256 further couples the processor 252 to a display
`controller 258, a mass memory 262, the modem or network
`interface 268, and an input/output (I/O) controller 264. The
`mass memory 262 may represent a magnetic, optical, mag-
`neto-optical, tape, and/or other type of machine-readable
`medium/device for storing information. For example, the
`mass memory 262 may represent a hard disk, a read-only or
`writable optical CD, etc. The display controller 258 controls
`in a conventional manner a display 260, which may repre-
`sent a cathode ray tube (CRT) display, a liquid crystal
`display (LCD), a plasma display, or other type of display
`device. The I/O controller 264 controls I/O device(s) 266,
`which may include one or more keyboards, mouse/trackball
`or other pointing devices, magnetic and/or optical disk
`drives, printers, scanners, digital cameras, microphones, etc.
`It will be appreciated that the digital processing system
`250 represents only one example of a system, which may
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`have many different configurations and architectures, and
`which may be employed with the present invention. For
`example, Macintosh and Intel systems often have multiple
`busses, such as a peripheral bus, a dedicated cache bus, etc.
`On the other hand, a network computer, which may be used
`as a digital processing device of the present invention, may
`not include, for example, a hard disk or other mass storage
`device, but may receive routines and/or data from a network
`connection, such as the modem or interface 268,
`to be
`processed by the processor 252. Similarly, a Web TV sys-
`tem, which is known in the art, may be considered to be a
`digital processing system of the present invention, but such
`a system may not include one or more I/O devices, such as
`those described above with reference to I/O device(s) 266.
`Additionally, a portable communication and data processing
`system, which may employ a cellular telephone and/or
`paging capabilities, may be considered a digital processing
`system which may be used with the present invention.
`In the system 250 shown in FIG. 2, the mass memory 262
`(and/or the memory 254) may store media (e.g., video,
`audio, movies, etc.) which may be processed according the
`present
`invention (e.g. by way of tracks). Alternatively,
`media data may be received by the digital processing system
`250, for example, via the modem or network interface 268,
`and stored and/or presented by the display 260 and/or I/O
`device(s) 266. In one embodiment, packetized media data
`may be transmitted across a data communication network,
`such as a LAN and/or the Internet, in accordance with tracks.
`On the other hand, the processor 252 may execute one or
`more routines to use a file with one or more tracks, or
`alternatively, to create one or more tracks, to process media
`(e.g., a pre-packaged movie, audio file, video file, etc.) for
`presentation or packetization according to the tracks. Such
`routines may be stored in the mass memory 262, the memory
`264, and/or another machine-readable medium accessible by
`the digital processin