336
`
`IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 3, JUNE 1997
`
`The Performance of TCP/IP for Networks with
`High Bandwidth-Delay Products and Random Loss
`
`T. V. Lakshman, Member, IEEE, and Upamanyu Madhow, Senior Member, IEEE
`
`Abstract— This paper examines the performance of TCP/IP,
`the Internet data transport protocol, over wide-area networks
`(WANs) in which data traffic could coexist with real-time traffic
`such as voice and video. Specifically, we attempt to develop a basic
`understanding, using analysis and simulation, of the properties
`of TCP/IP in a regime where: 1) the bandwidth-delay product of
`the network is high compared to the buffering in the network
`and 2) packets may incur random loss (e.g., due to transient
`congestion caused by fluctuations in real-time traffic, or wireless
`links in the path of the connection). The following key results
`are obtained. First, random loss leads to significant throughput
`deterioration when the product of the loss probability and the
`square of the bandwidth-delay product is larger than one. Second,
`for multiple connections sharing a bottleneck link, TCP is grossly
`unfair toward connections with higher round-trip delays. This
`means that a simple first in first out (FIFO) queueing discipline
`might not suffice for data traffic in WANs. Finally, while the
`recent Reno version of TCP produces less bursty traffic than
`the original Tahoe version, it is less robust than the latter when
`successive losses are closely spaced. We conclude by indicating
`modifications that may be required both at the transport and
`network layers to provide good end-to-end performance over
`high-speed WANs.
`
`Index Terms—Flow control, congestion control, error recovery,
`Internet, TCP/IP, transport protocols.
`
`I. INTRODUCTION
`
`but also for determining how TCP needs to be modified in
`the longer term.
`We study two versions of TCP: one is the popular Tahoe
`version developed by Jacobson in 1988 [11] (henceforth called
`TCP-tahoe); the other is the Reno version, which includes the
`fast retransmit option together with a method for reducing the
`incidence of slow start, suggested by Jacobson in 1990 [12]
`(we will refer to this as TCP-reno). We attempt to develop a
`basic understanding of these schemes by considering one-way
`traffic over a single bottleneck link with FIFO transmission.
`For LANs, the round-trip delay of the connection is small, so
`that the bandwidth-delay product could be much smaller than
`the buffering on the bottleneck link. We are more interested,
`however, in WANs with large round-trip delays, so that the
`buffering on the bottleneck link is typically of the same order
`of magnitude as, or smaller than, the bandwidth-delay product
`(this is what we mean by high bandwidth-delay products
`throughout this paper). The bottleneck link may be shared by
`several TCP connections. In addition, we also assume that each
`packets may be lost randomly even after obtaining service at
`the bottleneck link.
`Random loss is a simple model for a scenario of particular
`interest in the context of networks with multimedia traffic,
`where transient fluctuations in real time traffic may cause
`irregularly spaced losses for data traffic. This would occur,
`for instance, for both the UBR and ABR service classes [1]
`in ATM networks. The only difference is that for ATM ABR,
`each connection would have a time-varying available rate de-
`termined by feedback from the switches, so that most random
`losses would occur at the interface of the source to the network,
`since that is where the available rate would be enforced. In
`addition to serving as a model for transient congestion, we
`note that random loss on the Internet has been reported [3],
`where it is conjectured to occur due to a variety of reasons,
`including intermittent faults in hardware elements such as
`Ethernet/FDDI adapters, and incorrect handling of arriving
`packets by routers. Finally, with the anticipated emergence
`of mobile computing over heterogeneous networks with both
`wireless and wireline links, losses and time variations due
`to wireless links in the path of the connection can also be
`accommodated via a random loss model. Since our purpose
`is to obtain a fundamental understanding of TCP, none of the
`preceding situations are explicitly considered in this paper.
`However, as discussed in Section VI, the results here should
`provide a basis for further work on developing network level
`design guidelines for supporting TCP.
`One of the simplifications of the model used for our analysis
`is that two-way traffic (and the accompanying ack compression
`1063–6692/97$10.00 ª
`
`MOST existing data transfer protocols have been de-
`
`signed for local-area network (LAN) applications in
`which buffer sizes far exceed the bandwidth-delay product.1
`This assumption may not hold for the wide-area networks
`(WANs) formed by the interconnection of LANs using high-
`speed backbone networks. In addition, in the Internet of the
`future, data traffic will share the network with voice and video
`traffic. In this paper, we examine the impact of these changes
`on the performance of the most popular data transfer protocol
`in current use, TCP/IP. This is essential not only for network
`provisioning in the short term (since the rapid growth of Web
`applications has caused TCP traffic to grow correspondingly)
`
`Manuscript received June 20, 1995 revised February 25, 1997; approved by
`IEEE/ACM TRANSACTIONS ON NETWORKING Editor D. Mitra. This work was
`supported in part by the U.S. Army Research Office under Grant DAAH04-
`95-1-0246.
`T. V. Lakshman is with the High Speed Networks Research Dept.,
`Bell Laboratories, Lucent Technologies, Holmdel, NJ 07733 USA (e-mail:
`lakshman@research.bell-labs.com).
`U. Madhow is with the ECE Department and the Coordinated Science
`Laboratory, University of
`Illinois, Urbana,
`IL 61801 USA (e-mail:
`madhow@uiuc.edu).
`Publisher Item Identifier S 1063-6692(97)04489-0.
`1 The bandwidth-delay product is loosely defined to be the product of the
`round-trip delay for a data connection and the capacity of the bottleneck link
`in its path.
`
`1997 IEEE
`
`EX.1108.001
`
`DELL
`
`

`

`LAKSHMAN AND MADHOW: THE PERFORMANCE OF TCP/IP FOR NETWORKS WITH HIGH BANDWIDTH-DELAY PRODUCTS AND RANDOM LOSS
`
`337
`
`round-trip time. While we consider a similar system in Section
`V, our analysis is more detailed, taking explicit account of
`the buffer size and the bandwidth-delay product. Oscillatory
`behavior and unfairness toward connections with larger prop-
`agation delays have also been noticed in a previous analytical
`study of feedback-based congestion control [2]. Other analyses
`of flow control schemes include [20], [22], [23], but these
`references do not address the specific concerns raised here in
`any detail.
`The system model is described in Section II. Analytical and
`simulation results for the evolution of a single connection in
`the absence of random loss are given in Section III. Section IV
`considers the effect of random loss. Section V contains results
`for multiple connections with and without random loss. We
`give our conclusions in Section VI.
`
`II. SYSTEM MODEL
`We consider infinite data sources which always have packets
`to send, so that the units of data are maximum sized packets
`(in general, packet sizes in TCP may be variable). We consider
`a single bottleneck link with capacity
`
`[27]) is not considered. Feedback systems are notoriously
`difficult to analyze, so that even our simple model is not
`amenable to exact analysis. However, not only does our
`approximate analysis match simulation results for the idealized
`system model, but it also provides a close match to results for a
`detailed simulation that includes two-way traffic for multiple
`TCP-Reno connections over an ATM network (described in
`Section V).
`We obtain the following key results. Discussion of the
`implications of these results for system design is postponed
`to Section VI.
`1) While TCP-reno produces less bursty traffic than TCP-
`tahoe, it is much less robust toward “phase effects.”
`The latter term refers to unpredictability in performance
`resulting from very small differences in the relative tim-
`ings of packet arrivals for different connections sharing
`a link.
`2) Both versions of TCP appear to have significant draw-
`backs as a means of providing data services over mul-
`timedia networks, because random loss resulting from
`fluctuations in real-time traffic can lead to significant
`throughput deterioration in the high bandwidth-delay
`product regime. Roughly speaking, the performance is
`degraded when the product of the loss probability and
`the square of the bandwidth-delay product is large (e.g.,
`ten or more).
`3) For high bandwidth-delay products, TCP is grossly un-
`fair toward connections with higher propagation delays:
`for multiple connections sharing a bottleneck link, the
`throughput of a connection is inversely proportional to
`(a power of) its propagation delay.
`It is worth clarifying that random loss causes performance
`deterioration in TCP because it does not allow the TCP
`window to reach high enough levels to permit good link
`utilization. On the other hand, when the TCP window is
`already large and is causing congestion, random early drops of
`packets when the link buffer gets too full can actually enhance
`performance and alleviate phase effects [10].
`Early simulation studies of TCP-tahoe include [24], [26],
`[27]. Our model is similar to that used in [24], but the key
`differences between our paper and previous studies are that:
`1) the ratio of bandwidth-delay product to buffer size is much
`higher in our study and 2) the effect of random loss due
`to transient congestion (or other sources) is included. Thus,
`some of the undesirable features of TCP-tahoe which arise
`specifically for networks with high bandwidth-delay products
`(such as excessive buffering requirements and vulnerability to
`random loss) were not noticed in earlier studies. Furthermore,
`in contrast to previous studies, we place more emphasis on
`detailed analytical insight on the effects of various parameters
`on performance.
`The bias of TCP-tahoe against connections with large round-
`trip delays and against connections traversing a large number
`of congested gateways has been noticed in other studies of
`TCP-tahoe [8], [9], [26]. A heuristic analysis in [8] shows
`that, for multiple connections sharing a bottleneck link, the
`throughput of a connection is inversely proportional to its
`
`

`

`338
`
`IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 3, JUNE 1997
`
`started. Expiry of this timer is taken to signal packet loss.
`For each retransmission following a timer expiry, the timer
`value used is twice the previous timer value. Estimates of the
`round-trip time are obtained by measuring the round-trip time
`upon receipt of unambiguous acknowledgment (i.e., ignoring
`acknowledgment for retransmitted segments) and computing a
`weighted average of the old and new estimates. Refer to [15],
`[25] for a detailed description of round-trip time estimation.
`We will refer to a timer based on this estimate as a fine-grained
`timer, in order to distinguish it from the coarse-grained timers
`used in practice, which are typically multiples of 500 ms. In
`order to prevent a needlessly lengthy stoppage of transmission
`upon expiry of a coarse-grained timer, most current versions
`of both TCP-tahoe and TCP-reno incorporate a fast retransmit
`option:
`if the number of duplicate acknowledgments (i.e.,
`multiple acknowledgment with the same “next expected”
`packet number
`
`

`

`LAKSHMAN AND MADHOW: THE PERFORMANCE OF TCP/IP FOR NETWORKS WITH HIGH BANDWIDTH-DELAY PRODUCTS AND RANDOM LOSS
`
`339
`
`avoidance ends. The value of
`
`

`

`340
`
`IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 3, JUNE 1997
`
`Fig. 1. Window and buffer evolution for a single connection: Two slow
`starts. Prop. delay = 1 ms; b =0.1.
`
`size as follows. Define the integer
`
`

`

`LAKSHMAN AND MADHOW: THE PERFORMANCE OF TCP/IP FOR NETWORKS WITH HIGH BANDWIDTH-DELAY PRODUCTS AND RANDOM LOSS
`
`341
`
`sult in excellent agreement with the throughput obtained by
`simulations.
`Case 1
`
`

`

`342
`
`IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 3, JUNE 1997
`
`TABLE II
`LINK UTILIZATION AS A FUNCTION OF
`NORMALIZED BUFFER SIZE ( = 100;  = 1)
`
`C. Throughput Computation and Numerical Results
`Due to the periodic evolution, the long-run average through-
`puts for both TCP-tahoe and TCP-reno are equal to the average
`throughputs in a cycle, and are given by
`
`

`

`LAKSHMAN AND MADHOW: THE PERFORMANCE OF TCP/IP FOR NETWORKS WITH HIGH BANDWIDTH-DELAY PRODUCTS AND RANDOM LOSS
`
`343
`
`window size at the time of loss is
`
`

`

`344
`
`IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 3, JUNE 1997
`
`TABLE III
`LINK UTILIZATION AS A FUNCTION OF LOSS PROBABILITY q FOR
` = 100,  = 1, AND TWO VALUES OF

(

= 0:8 AND

= 0:2)
`
`TABLE IV
`LINK UTILIZATION AS A FUNCTION OF q( )2 FOR THREE
`DIFFERENT VALUES OF BANDWIDTH-DELAY PRODUCT (

= 0:8)
`
`the throughput, is dominated by the effect of random losses.
`This throughput is much smaller than the lossless throughput
`and is insensitive to the value of
`
`

`

`LAKSHMAN AND MADHOW: THE PERFORMANCE OF TCP/IP FOR NETWORKS WITH HIGH BANDWIDTH-DELAY PRODUCTS AND RANDOM LOSS
`
`345
`
`analytically whether such synchronization would always occur,
`but the basic observation on which our analysis is based, which
`is that the window size grows more slowly for connections
`with higher round trip delays, should apply even if there were
`instances of evolution without synchronization.
`
`A. Intuitive Explanation for the Bias
`An approximate expression for the instantaneous throughput
`for connection
`
`

`

`346
`
`IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 3, JUNE 1997
`
`earlier. The analysis is approximate for several reasons: i) an
`exact analysis of multiple connections with different propa-
`gation delays is not available even for fixed windows, which
`makes it necessary to approximate the queueing delays
`
`

`

`LAKSHMAN AND MADHOW: THE PERFORMANCE OF TCP/IP FOR NETWORKS WITH HIGH BANDWIDTH-DELAY PRODUCTS AND RANDOM LOSS
`
`347
`
`Substituting (31) in (29), we obtain
`
`

`

`348
`
`IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 3, JUNE 1997
`
`TABLE V
`RELATIVE THROUGHPUTS FOR TWO TCP-TAHOE CONNECTIONS WITH
`DIFFERENT PROPAGATION DELAYS FOR  = 100 AND BUFFER SIZE B = 80
`
`TABLE VI
`RELATIVE THROUGHPUTS FOR TWO CONNECTIONS WITH 1 = 80
`MS, 2 = 40 MS,  = 96 000 CELLS/S (DS-3 LINK RATE)
`
`D. Numerical Results
`For TCP-tahoe, analysis and simulation results are compared
`for the simple model described in Section II. Table V gives
`the relative throughputs
`
`

`

`LAKSHMAN AND MADHOW: THE PERFORMANCE OF TCP/IP FOR NETWORKS WITH HIGH BANDWIDTH-DELAY PRODUCTS AND RANDOM LOSS
`
`349
`
`is, for each context of interest, to translate these into specific
`recommendations, and to provide systematic design techniques
`for arriving at these recommendations. Especially interesting
`is the question of how best to support TCP over the ATM ABR
`and UBR service classes, since that involves adaptation at both
`the network and transport layers. Another important area for
`future research is the development of an alternative dynamic
`window mechanism which addresses some of the shortcomings
`of TCP while preserving its decentralized nature. Possible
`improvements might be better congestion avoidance via more
`sophisticated processing of round-trip delay estimates, and the
`use of selective acknowledgments to improve the performance
`in the presence of random loss.
`
`APPENDIX
`We give the details of the approximation for the throughput
`with loss, taking as our starting point (27) in Section IV.
`We consider only the case
`
`connection set-up and enforced at switches and routers
`using per connection queueing [6], [21]. Since admin-
`istering the resources allocated to every best effort
`connection may be excessively expensive, a more fea-
`sible alternative might be to allocate and administer
`resources for an entire traffic class. In such a situation,
`the unfairness we have pointed out would persist if
`TCP were supported over the ATM UBR traffic class.
`However, if TCP is supported over the ABR traffic class,
`the time-varying rate available to each connection is
`determined at the network level and is administered at
`the source, so that different TCP connections should be
`isolated from each other to a large extent even if they
`share the same network buffers.
`4) In addition to causing vulnerability to random loss,
`the fact that loss is the sole means of feedback used
`by TCP leads to excessive delays. This is because,
`in networks with high utilizations,
`the window size
`for a TCP connection would keep increasing after the
`bottleneck link is fully utilized, until in fact there is a
`buffer overflow leading to a loss. The delay and loss
`performance would improve significantly if we instead
`used a scheme that tries to maintain a window size which
`is just large enough to achieve a high link utilization.
`A scheme such as DECbit [22] attempts to do this
`using explicit feedback from the switches, and similar
`schemes are worth pursuing, especially because Explicit
`Congestion Notification is incorporated as an option for
`ATM networks. Note, however, that the DECbit scheme
`in particular shares with TCP the problem of unfair-
`ness toward connections with longer propagation delays.
`Another possibility is a more sophisticated processing
`of round-trip time estimates similar to the approach
`taken in [18], [19]. This is certainly attractive, since
`it avoids the need for explicit feedback. However, if
`the round-trip delays can change substantially without
`changes in the load on the path of the connection
`(e.g., because processing delays at nodes depend on
`the load on the operating system, or because of delays
`due to handoffs for mobile computing applications),
`then adaptation based on delay processing might be less
`robust than adaptation based on loss or explicit feedback.
`In addition,
`if different connections are not
`isolated
`from each other in terms of their use of bandwidth and
`buffering in the network, then a connection that is more
`aggressive about obtaining bandwidth by increasing its
`rate until
`there is a loss would be at an advantage
`over connections that process round-trip delays to avoid
`congestion. Thus, changes at the transport layer must
`either be adopted universally, or must go hand in hand
`with network layer controls that guard against greedy
`connections.
`In summary, while we have identified several shortcomings
`of TCP, we have also mentioned possible means of obtain-
`ing good performance via network level solutions, such as
`isolating connections from each other and providing enough
`buffering to hide excessively fast time variations in available
`link capacity from TCP. An important topic for future research
`
`

`

`350
`
`IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 5, NO. 3, JUNE 1997
`
`average throughput:
`
`