throbber
INCREASINGTCPTHROUGHPUTBYUSINGANEXTENDED
`ACKNOWLEDGEMENTINTERVAL
`Athesispresentedto
`TheFacultyof
`TheCollegeofArtsandSciencesofOhioUniversity
`InPartialFul(cid:12)llment
`oftheRequirementsfortheDegree
`MasterofScience
`StacyR.Johnson
`June 
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 001
`
`

`

`INCREASINGTCPTHROUGHPUTBYUSINGANEXTENDED
`ACKNOWLEDGEMENTINTERVAL
`BY
`STACYR.JOHNSON
`Thisthesishasbeenapproved
`fortheDepartmentofMathematics
`andtheCollegeofArtsandSciencesby
`ShawnD.Ostermann
`AssistantProfessorofComputerScience
`HaroldMolineu
`Dean,CollegeofArtsandSciences
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 002
`
`

`

`Tomyfamily
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 003
`
`

`

`ABSTRACT
`Johnson,StacyR..M.S.,OhioUniversity,June .IncreasingTCPThroughput
`byUsinganExtendedAcknowledgementInterval.MajorProfessor:Dr.ShawnD.
`Ostermann.
`TheTransmissionControlProtocol(TCP)isawidelyusednetworkprotocolthat
`isusuallylayeredovertheInternetProtocol(IP).IPisnotreliable;itdoesnotinsure
`datadelivery.However,TCPisreliable,andinsuresdatadeliverythroughtheuseof
`acknowledgementsandretransmissions.
`ThereceivingsideofaTCPconnectioncurrentlyacknowledgesatleasteveryother
`datablock(segment)thatitreceives.Someoftheseacknowledgementsmaybesu-
`per(cid:13)uous.AlargeportionofaTCPconnection'scostistheoverheadofprocessinga
`newpacketasitarrives.Consequently,itispossiblethatextendingtheacknowledge-
`mentintervalmayincreasethroughputbyrequiringlessacknowledgementoverhead
`perconnection.
`WeimplementedextendedacknowledgementintervalsinUNIX.BSDcompat-
`iblenetworkcode.WecompiledthecodeonaSunIPCworkstationrunningSunOS.
`Totestourhypothesis,weconductedthroughputtestsusingmachinesonthe
`samelocalnetwork,anearnetwork,andadistantnetwork.Analysisofthedata
`revealedthatextendingtheacknowledgementintervaldoesincreasethroughputbya
`statisticallysigni(cid:12)cantamountinthelocalandnearnetworkcases.Inparticular,in
`thelocalcasethethroughputwasincreasedbyasmuchaspercentinsomecases.
`ItisprobablethatcongestionontheInternetandtheresultingpacketlossprevented
`theoverallincreaseinthroughputonthenon-localconnections.
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 004
`
`

`

`ACKNOWLEDGMENTS
`Thankstomyfamilyandmyfriends;yoursupportoverthepastyearisgreatly
`appreciated.SpecialthankstomyfellowmembersofOhioUniversity'sInternetwork-
`ingResearchGroupforthelivelydiscussions.Also,thankstoDr.Ostermannfor
`yourideas,patience,andforintroducingmetothewideworldofnetworking.
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 005
`
`

`

`DISCARDTHISPAGE
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 006
`
`

`

`TABLEOFCONTENTS
`ABSTRACT::::::::::::::::::::::::::::::::::::
`LISTOFTABLES:::::::::::::::::::::::::::::::::
`LISTOFFIGURES::::::::::::::::::::::::::::::::
`INTRODUCTION:::::::::::::::::::::::::::::::
` .
` . TCP/IPOverview::::::::::::::::::::::::::::
` .AcknowledgementsinTCP::::::::::::::::::::::::
` .. TCPProcessingOverhead::::::::::::::::::::
` ..CurrentTCPAcknowledgementImplementation::::::::
` .. TimingTCPSegments::::::::::::::::::::::
` ..DelayedAcknowledgementsandtheRoundTripTime
`::::
` . WindowSizeandThroughputinTCP:::::::::::::::::
` .ExtendedAcknowledgementIntervals::::::::::::::::::
`IMPLEMENTATION:::::::::::::::::::::::::::::
`.
`. Environment::::::::::::::::::::::::::::::::
`.
`ImplementationDetails::::::::::::::::::::::::::
`..
`tcpvar.h:TrackingFields::::::::::::::::::::
`..
`tcpinput.c:::::::::::::::::::::::::::::
`..
`tcpoutput.c::::::::::::::::::::::::::::
`..
`tcpsubr.c:::::::::::::::::::::::::::::
`..
`tcptimer.c:::::::::::::::::::::::::::::
`
`
`Page
`
`
`
`
`
` 
`
` 
` 
` 
` 
` 
`
`
`
`
`
`
`
`
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 007
`
`

`

` .EXPERIMENTALRESULTS:::::::::::::::::::::::::
` ..
`StatisticalAnalysis::::::::::::::::::::::::
` . LocalNetwork:::::::::::::::::::::::::::::::
` . . Gatheringdata::::::::::::::::::::::::::
` . .ExperimentswithakWindowSize
`::::::::::::::
` . . Experimentswitha kWindowSize::::::::::::::
` . .Experimentswitha kWindowSize::::::::::::::
` .NearNetwork:::::::::::::::::::::::::::::::
` .. ExplanationofResults::::::::::::::::::::::
` . DistantNetwork:::::::::::::::::::::::::::::
` . . ExplanationofResults::::::::::::::::::::::
`.CONCLUSIONS::::::::::::::::::::::::::::::::
`. Overview::::::::::::::::::::::::::::::::::
`.Conclusions::::::::::::::::::::::::::::::::
`. RecommendationsforFurtherResearch:::::::::::::::::
`BIBLIOGRAPHY:::::::::::::::::::::::::::::::::
`APPENDIX
`A.HOWTOREADATCPTIMESEQUENCEGRAPH:::::::::::
`
`Page
`
`
`
`
`
`
` 
` 
`
`
`
`
`
`
`
`
`
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 008
`
`

`

`
`Page
` 
`
`
`
`LISTOFTABLES
`Table
`:::::::::::::
` . TheE(cid:11)ectofTCPWindowSizeonThroughput
`tcpinput.cTrackingVariables
`:::::::::::::::::::::::
`.
`tcpoutput.cTrackingVariables:::::::::::::::::::::::
`.
`APPENDIX
`Table
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 009
`
`

`

`LISTOFFIGURES
`Figure
` . ProtocolLayeringontheInternet::::::::::::::::::::::
` .TCPHeaderFormat:::::::::::::::::::::::::::::
`. TCPResearchVariableStructure::::::::::::::::::::::
`.ExtendingtheAcknowledgementInterval
`:::::::::::::::::
` . RawThroughputCurve:::::::::::::::::::::::::::
` .AdjustedThroughputCurve
`::::::::::::::::::::::::
` . LocalNetworkThroughput-kBu(cid:11)erSize::::::::::::::::
` .AverageNumberofAcknowledgementsperConnection::::::::::
`kWindowSizeLeadstoAck-Every-OtherConditions::::::::::
` .
` .LocalNetworkThroughput- kBu(cid:11)erSize
`:::::::::::::::
` .E(cid:11)ectofBu(cid:11)erSizeonLocalNetworkThrougput:::::::::::::
` .LocalNetworkThroughput- kBu(cid:11)erSize
`:::::::::::::::
` . ExtendedAcknowledgementIntervalof- kWindowSize:::::::
` . AcknowledgementIntervalof(Default)- kWindowSize:::::::
` . NearNetworkThroughput
`:::::::::::::::::::::::::
` . DistantNetworkThroughput::::::::::::::::::::::::
`A. ExampleTCPTimeSequenceGraph::::::::::::::::::::
`
`
`Page
`
`
`
`
`
`
`
`
` 
` 
` 
` 
` 
`
`
`
`
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 010
`
`

`

`
` .INTRODUCTION
` . TCP/IPOverview
`Anetworkprotocolisasetofrulesthatcomputersusewhenexchangingdata
`overacomputernetwork.TCPandIParethetwoprotocolsthatformthebasis
`ofsharinginformationovertheInternet.Aprotocolisreliableifitguaranteesdata
`delivery.Anunreliableprotocoldoesnotguaranteedatadelivery.Aprotocolis
`connectionlessifeachpacketsentovertheprotocolistreatedindependently[Com].
`Withaconnectionlessprotocol,eachpacketmaytakeadi(cid:11)erentpathtoreachits
`destination,andthepacketsmayarriveinanyorder.
`Dataissplitintopiecesbeforeitissentoveranetwork.Beforetransmittingthe
`data,theprotocoladdsaheaderthatincludesthesourceanddestinationofthedata.
`OnanEthernet,theresultingunitiscalledaphysicalframe[Com].Atahigher
`protocollevel,eachunitmaybecalledadatagramorasegment.
`Networkprotocolsareoftenlayered.Acompletepacketincludingtheheaderand
`datafromoneprotocolmaybecontained,orencapsulated,withinthedatasection
`ofanotherprotocol.Ahigh-levelprotocolmaybeencapsulatedwithinalow-level
`protocol,whichmayitselfbeencapsulatedwithinanotherprotocol.Eachprotocol
`layeraddsusefulfeaturestotheonebelowit.
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 011
`
`

`

`
`TheInternetProtocol(IP)isanunreliable,connectionlessprotocolthatdeliv-
`ersdatabetweencomputersovertheInternet.EachunitsentusingIPiscalleda
`datagram.
`TheTransmissionControlProtocol(TCP)isareliable,connection-oriented[Com]
`protocolthatdeliversdatabetweenapplications.TCPcanbeusedtodeliverdatabe-
`tweenapplicationsonthesamecomputer,ortodi(cid:11)erentcomputersoveraninternet.
`AlthoughTCPisageneralpurposeprotocolwhichcanbeusedwithseveraldelivery
`systems,ontheInternetitsdataunits,calledsegments,areencapsulatedwithinIP
`datagrams[Com]asshownin(cid:12)gure . .EventhoughTCPisencapsulatedwithin
`theconnectionlessIPprotocol,itisconnection-oriented[Com]becausehostsmust
`negotiatetoopenandcloseaTCPconnection.
`Figure . ProtocolLayeringontheInternet
`WhenusingTCPontheInternet,dataisencapsulatedwithinaTCPsegment,anIP
`datagram,aphysicalframe,andthentransmitted.
`
`Physical network frame
`
`TCP segment
`
`IP datagram
`
`(e.g. Ethernet frame)
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 012
`
`

`

` 
` .AcknowledgementsinTCP
`Asmentionedpreviously,IPisanunreliableprotocol.IPhasa\best-e(cid:11)ort"de-
`liveryscheme,butdoesnotguaranteethedeliveryofanydata.Tocompensatefor
`this,TCPisreliable;itguaranteesthedeliveryofdatathroughtheuseofacknowl-
`edgementsandretransmissions..
`EachbyteofdatathatissentthroughaTCPconnectionisacknowledgedusing
`apositiveacknowledgement,whichmeansthatthereceivingTCPacknowledgesdata
`thatitreceives .IfthereceivingTCPdoesnotacknowledgedatawithinacertain
`timelimit(seesection .. )thesendingTCPwillretransmitthedata.TCPac-
`knowledgementsarealsocumulative;ahostacknowledgingreceiptofthenthbyteis
`alsoimplicitlyacknowledgingthereceiptofallbytespreviouston.
`EachsideofaTCPconnectionchoosesaninitialsequencenumberforthe(cid:12)rst
`byteofdatathatTCPsendsoverthenetwork.Subsequentdatabytesarenumbered
`sequentiallystartingatthisnumber.ThereceivingTCPmustacknowledgereceiptof
`everydatabytesentthroughaTCPconnection,atleastcumulatively.Thereceiving
`TCPacknowledgesthedatabysendinganacknowledgement(ACK)whichcontains
`thesequencenumberofthenextbyteitexpectstoreceive.Forexample,ifasegment
`arrivescontainingbyteswiththesequencenumbers -,thereceivingTCP
`canacknowledgeallofthebytesinthesegmentbysendinganacknowledgement
`containingthesequencenumber .
`TCPconnectionsarefull-duplex,whichmeansthatdatacan(cid:13)owinbothdirections
`simultaneously[Com].The(cid:12)eldsthatareusedforacknowledgementsarecontained
`intheheaderoftheTCPsegment(see(cid:12)gure .),whichallowsTCPtosendan
`acknowledgementanddatainthesamesegment.Thisisknownaspiggybackingan
`acknowledgementonadatasegment.
` Incontrast,someprotocolsuseanegativeacknowledgement;thereceiverwillsendamessageif
`itdoesnotreceiveexpecteddata.
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 013
`
`

`

`16
`
`24
`Destination Port
`
`31
`
`0
`
`4
`Source Port
`
`10
`
`Data Offset
`
`Reserved
`Checksum
`
`Sequence Number
`Acknowledgement Number
`Flags
`
`
`Figure .TCPHeaderFormat[Pos ]
`ThereisspacereservedforanacknowledgementineachTCPsegment.Ahostmay
`senddataandanacknowledgementinthesamepacketbysettingtheacknowledge-
`ment(cid:13)agand(cid:12)llingtheacknowledgement(cid:12)eldwiththe bitsequencenumberof
`thenextbyteitexpectstoreceive.
` .. TCPProcessingOverhead
`AsimpleversionofTCPacknowledgeseverysegmentsent.Thus,foreverydata
`segmentsentoverthenetwork,thereisalsoanacknowledgementsegment.Most
`ofthecostofmaintainingaTCPconnection,intermsofcomputerresources,is
`duetotheoverheadofdealingwithincomingpackets[Cla].RecallthatTCP
`acknowledgementsarecumulative;thosesentneareachothermayberedundant.
`Eliminatingsomeoftheredundantacknowledgementswillhelptoalleviateconnection
`overhead.
`
`Window
`Urgent Pointer
`Padding
`
`Options
`Data (variable length)
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 014
`
`

`

` 
` ..CurrentTCPAcknowledgementImplementation
`Thecurrentspeci(cid:12)cationofTCP[Bra ]recommendsdelayedacknowledgements.
`Speci(cid:12)cally,afterreceivingasegment,thereceivingTCPmaydelaysendingthe
`acknowledgementforuptoms.BSD-basedimplementationsofTCPwilldelay
`sendingtheacknowledgementforuptoms[WS ].Ifthehosthasoutgoingdata
`forthesameconnection,TCPcanpiggybacktheacknowledgementonthenextdata
`segment.Thisisusefulforcuttingdownonthenumberofsegmentsininteractive
`connectionssuchasatelnetconnection,whereeachcharactertypedisimmediately
`echoedbacktothesender.Furthermore,accordingtothestandard,thereceiving
`TCPmustsendanacknowledgementforatleasteverysecondsegment[Bra ].
` .. TimingTCPSegments
`Asmentionedpreviously,thereceivingTCPusespositiveacknowledgementsto
`signalreceiptofdatasegments.Ifadatasegmentislostordamaged,thereceiving
`TCPwillnotacknowledgeit.Inthiscase,thesendingTCPneedsamechanismto
`recognizewhentoresendasegment.ThesendingTCPmaintainsaretransmission
`timeoutvalue(RTO),whichistheamountoftimethatTCPwillwaitbeforere-
`sendingasegment[Ste ].
`Theroundtriptime(RTT)ofasegmentistheelapsedtimebetweensendinga
`segmentandreceivinganacknowledgementthatcoversthesegment.TheRTOvalue
`isbasedontheroundtriptimesofTCPsegments(cid:13)owingoverthegivenconnection.
`TheRTTofsegments(cid:13)owingoveraconnectioncanvarywidely[Jac],sotheRTTof
`individualsegmentsarecombinedthroughasmoothingalgorithm[Jac]toprovide
`abetterestimateoftheRTTforsubsequentsegments.
`TCPhastwotimersrunningatalltimes:amsfasttimerandamsslow
`timer.TCPmeasurestheRTTofasegmentinslowtimerticks,agranularityof
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 015
`
`

`

` 
`ms[WS ].TheTCPVegasresearchproject[BOP ]claimstohavegottenhigher
`throughputbyusingUNIXtimestampswithamillisecondgranularity,asopposed
`totheslowtimertick.
`TCPtimesonlyonesegmentatatime[Ste ].Consequently,TCPonlytimes
` segmentpereverynsegments,wherenisthenumberofsegmentsintransitata
`giventime.ThesamplingoftheRTTisdependentonthenumberofsegmentsin
`transitsimultaneously[JBB ].Severalsuggestionshavebeenmadeforimproving
`theaccuracyoftheRTTmeasurements.Forexample,ifTCPtimedeverysegment,
`itcouldapplytheRTTtothelastsegmentcoveredbytheacknowledgement.This
`wouldincreasetheRTTsamplingrate,andwouldresultinamoreaccurateRTT.
`SuchaschemeisproposedinTCPVegas[BOP ].TCP'stimestampoption[JBB ]
`o(cid:11)ersanotherwaytogetamoreaccurateRTT.Whenusingthisoption,TCPin-
`cludesatimestampineachsegmentthatissent.Thereceiverechoesthistimestamp
`intheacknowledgementsegment.ThesenderisabletocalculateaRTTforeach
`acknowledgementitreceives[JBB ].Currently,TCPcanonlycalculateaRTTfor
`anacknowledgementofatimedsegment(onesegmentperwindow)[Ste ].
` ..DelayedAcknowledgementsandtheRoundTripTime
`Delayedacknowledgementsa(cid:11)ecttheRTTbecauseofthewaythatTCPcurrently
`timesthesegments(seesection .. ).AssumethatthesendingTCPsendsthe
`segmentitistiming,segmentn,followedbysegmentn+ .IfthereceivingTCP
`delaysacknowledgementofsegmentn,thenthesendingTCPwillcalculatealonger
`RTTforthesegmentnwhenitreceivestheacknowledgementofsegmentn+ [Ste ].
`TCP'stimestampoptionmaybeusedinconjunctionwithdelayedacknowledge-
`ments.Ifdelayedacknowledgementsarebeingused,thereceivingTCPechoesthe
`AUNIXtimestampisanintegerwhichrepresentsthepresenttimeasthenumberofseconds
`sinceJanuary , .
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 016
`
`

`

` 
`earliesttimestampitreceivedthatithasnotyetacknowledged.ThesendingTCP
`thenautomaticallytakesintoaccountthetimeforthedelayedacknowledgements
`[JBB ].Whenusingdelayedacknowledgementsasweareinthisresearch,alonger(in-
`correct)RTTmeasurementisvital.IftheRTTisaccurate,TCPmayconstruethe
`longerRTTcausedbythedelayedacknowledgementstomeanlostsegments,and
`startretransmittingsegmentsunnecessarily[Bra ].
` . WindowSizeandThroughputinTCP
`EachsideofaTCPconnectionhasabu(cid:11)erforincomingdata.Theamountof
`bu(cid:11)erspaceavailableiscalledthewindowsize.Datasentthroughtheconnectionis
`heldinthereceivingTCP'sbu(cid:11)ersuntilitcanbedeliveredtotheapplicationreading
`fromtheconnection.Theavailablewindowsizewillvarythroughouttheduration
`oftheconnection,dependingonhowmuchdatahasarrivedatthehostandisstill
`waitingtobeprocessed.
`Ifyouvisualizetheconnectionbetweentwohostsasapipeline,thenthesizeof
`thebu(cid:11)ersonbothsidesoftheconnectionlimithowmuchinformationcanbeinthe
`pipelineatonetime.Theamountofinformationinthepipelinemustneverexceed
`thereceivingTCP'sbu(cid:11)erspace.TCPwillattemptto(cid:12)lltheconnectionbetween
`thehostswithasmuchdataaspossible.Ingeneral,asthebu(cid:11)ersizegrows,sodoes
`theamountofdatathatcanbeintransitatagiventime.
`Throughputisameasureoftheamountofdata(cid:13)owingthroughaconnectionper
`unittime.Ingeneral,onedesirestomaximizethethroughputofagivenconnection.
`TheTCPwindowsizea(cid:11)ectsthroughputbylimitingtheamountofdatathatcanbe
`intransitatagiventime.Forexample,ifthewindowsizeofthereceivinghostis
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 017
`
`

`

` 
`small,thenthesendinghostwillhavetopauseperiodicallythroughouttheduration
`oftheconnectiontoallowthewindowtoopenagain[Ste ].
`IfthesendingTCPmanagesto(cid:12)llthereceivingTCP'sbu(cid:11)ers,thereceiving
`TCPwillsendazerowindowadvertisementwhichcausesthesendingTCPtostop
`transmittingdataovertheconnection[Ste ].ThesendingTCPwillsetapersist
`timer;ifithasnotreceivedawindowupdatewhenthetimerexpires,itwillsenda
`messagetothereceivingTCPtoseeifawindowupdatehasbeenlost[Ste ].During
`thetimethatthesendingTCPwaitsforthereceivingTCPtoemptyinformationfrom
`itsbu(cid:11)ers,thepipelinebetweenthehostsmayemptyout.Oncebu(cid:11)erspacebecomes
`available,itwilltaketimetore(cid:12)llthepipewithdataandhavetheconnectionrunning
`at\fullcapacity"again.Table . showsthetheoreticalmaximumthroughputfor
`variouswindowsizes,ascalculatedbythebandwidth-delayproductequation[Ste ].
`RoundTripTime(ms)
`local
`cross-campusKentStateUofKentuckySunsiteCA
`Bu(cid:11)erSize
`(.ms)
`(.ms)
`(ms)
`( ms)
`(ms)
`k
` .
`.
`.
`.
`.
` k
`.
` .
`.
`.
`.
` k
` .
`. 
`.
`.
`. 
`k
`.
` .
` .
`.
`.
`Table . TheE(cid:11)ectofTCPWindowSizeonThroughput
`Foreachhost/RTTandbu(cid:11)ersizelisted,throughputisshowninMB/s.Themaxi-
`mumpossiblethroughputincreasesastheTCPwindowsize(bu(cid:11)ersize)increases.
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 018
`
`

`

` 
` .ExtendedAcknowledgementIntervals
`BasedonourobservationsofTCP,wehypothesizethatextendedacknowledge-
`mentintervalsmayallowforgreaterthroughputonaTCPconnection.TCPcur-
`rentlyrequiresahosttoacknowledgeatleasteveryothersegment[Bra ].Thisisan
`acknowledgementintervaloftwo;twosegmentsmaybereceivedbeforeanacknowl-
`edgementmustbesent.Itispossiblethatextendingtheacknowledgementinterval
`mayincreasethroughputbyrequiringlessacknowledgementoverheadperconnection.
`Theresearchpresentedintherestofthispaperteststhee(cid:11)ectsofextended
`acknowledgementintervalsonthethroughputoflocalandnon-localTCPconnections.
`Chapterfocusesonthenecessaryoperatingsystemchangestoimplementextended
`acknowledgementintervals.Chapter describestheexperimentsperformedwith
`themodi(cid:12)edTCP.Chapterexaminestheresultsoftheexperimentsandgives
`suggestionsforfurtherresearch.
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 019
`
`

`

`
`.IMPLEMENTATION
`. Environment
`Wecompiledanexperimentaloperatingsystem(kernel)onaSunIPCworkstation
`(netipc)runningSunOS. .,fromSunMicrosystems.Wemadechangesto.BSD
`compatiblepublicdomainnetworkingcode,andcompiledthenetworkingcodeinto
`theSunOS. .operatingsystem.TheworkstationisattachedtoalocalEthernet
`network.
`.
`ImplementationDetails
`Fewchangesareneededtoimplementextendedacknowledgementintervalsin
`TCP.Mostofthechangesthatwemadetrackthee(cid:11)ectsoftheextendedacknowl-
`tcpinput.c,tcpoutput.c,tcpsubr.c,
`edgementintervals.Thea(cid:11)ected(cid:12)lesare:
`tcptimer.c,andtcpvar.h.Thefollowingsectionsdescribethechangesmadetoeach
`ofthe(cid:12)les.
`..
`tcpvar.h:TrackingFields
`EachTCPconnectionisassociatedwithaTCPcontrolblockthatholdsinforma-
`tionabouttheconnection.TCPkeepstrackofseveralstatistics.Forexample,TCP
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 020
`
`

`

`
`tracksthenumberofbytessent,thenumberofbytesreceived,andthenumberof
`duplicatepacketsreceived.However,TCPdoesnottracktheseonaper-connection
`basis.Weaddedanewstructurethatcontainsper-connectiontrackingdata((cid:12)g-
`ure. ).TheTCPcontrolblockhasapointertothisstructure.Weplacedcode
`toupdatethesetrackingvariablesnexttowheretheSunOStrackingvariableswere
`maintainedinthebasicTCPcode.
`ThenewversionofTCPalsokeepstrackofthenumberoftimestheextended
`intervalcodeiscalledduringthelifetimeofeachTCPconnection(variablestinc,
`tackdone,andttimeackin(cid:12)gure. ).
`#ifdefTCPEXP
`#defineMT_TCPE
`
`/*mbuftypeforourexperimentalvariables*/
`/*thisshouldbeinmbuf.h
`*/
`/*Thisstructureholdsthetcpexperimentalfields/statistics.*/
`/*ApointertothisstructureislocatedintheTCPcontrolblock.*/
`structtcpe{
`u_long
`t_acknowcnt;
`/*countdowntosettingacknowflag*/
`u_long
`t_inc;
`/*numtimesacknowcntincreased*/
`u_long
`t_ackdone;
`/*numtimesacknowreachedlimit*/
`u_long
`t_timeack;
`/*numtimestimedoutandacked*/
`u_long
`t_sndbyte;
`/*databytessent*/
`u_long
`t_sndrexmitpack;
`/*retransmittedpackets*/
`u_long
`t_sndrexmitbyte;
`/*databytesretransmitted*/
`u_long
`t_sndacks;
`/*ack-onlypacketssent*/
`u_long
`t_rcvbyte;
`/*bytesreceivedinsequence*/
`u_long
`t_rcvdupbyte;
`/*duplicate-onlybytesreceived*/
`u_long
`t_rcvpartdupbyte;
`/*dup.bytesinpart-dup.packets*/
`u_long
`t_rcvackpack;
`/*acksreceived*/
`u_long
`t_rcvackbyte;
`/*bytesackedbyrcvdacks*/
`u_long
`t_rcvdupack;
`/*duplicateackpacketsreceived*/
`};#endif
`Figure. TCPResearchVariableStructure
`Theexperimentalvariablesstructurehousesthevariablesusedinthisresearch.The
`TCPcontrolblockstructurehasapointerleadingtothisstructure.
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 021
`
`

`

`
`tcpinput.c
`..
`Themostextensivelyaltered(cid:12)leintheimplementationistcpinput.c.Thechanges
`totcpinputcomprisethebulkoftheextendedacknowledgementintervalimplemen-
`tation.Theheartofthecodeisshownin(cid:12)gure..
`if(tcp_nodelack==){
`(tp->t_exp->t_acknowcnt)++;
`(tp->t_exp->t_inc)++;
`if(tp->t_exp->t_acknowcnt==TCPE_ACKNOWCNT){
`tp->t_flags|=TF_ACKNOW;
`tp->t_exp->t_acknowcnt=;
`(tp->t_exp->t_ackdone)++;
`}elsetp->t_flags|=TF_DELACK;
`}elsetp->t_flags|=TF_ACKNOW;
`Figure.ExtendingtheAcknowldegementInterval
`Thispieceofcodeintcpinput.cchecksthenumberof\outstanding"segmentsto
`seeiftheextendedacknowledgementintervallimithasbeenreached.Ifthelimithas
`beenreached,itsetsa(cid:13)agtoacknowledgethesegment.
`Thispieceofcodeisplacedinthedodatasectionoftcpinput,whichprocessesan
`incomingsegmentandarrangesforanacknowledgmentofthesegment,ifnecessary.
`Ifdelayedacknowledgementsarenotenabled(theyareenabledbydefault),a(cid:13)ag
`issettoacknowledgethepacket.Ifdelayedacknowledgementsareenabled,thenthree
`thingsoccur.First,the(cid:12)eldthatcountsthenumberofoutstandingacknowledgements
`(tacknowcnt)isincremented.Second,thetincisincrementedtoshowthatthe
`outstandingacknowledgements(cid:12)eldwasincremented..Third,TCPdecideswhether
`thecurrentsegmentneedstobeacknowledgedimmediatelybycomparingthenumber
`ofoutstandingacknowledgementstothepresetlimit.
`Ifthenumberofoutstandingacknowledgementsreachesthepresetlimit
`(TCPEACKNOWCNT),thenthe(cid:13)agissettoacknowledgethesegmentimmediately.The
`outstandingacknowledgements(cid:12)eld(tacknowcnt)isreset,andthetrackingvariable
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 022
`
`

`

`
`tackdoneisincrementedtoshowthattheacknowledgementwassentbecausethe
`numberofoutstandingacknowledgementsmetthepresetlimit.
`Thecodethatsetstheinitialthresholdvalueforoutstandingacknowledgements
`(TCPEACKNOWCNT)isalsocontainedintcpinput.c.Settingtheintervaltoevokes
`the\ack-every-other"behaviorofthestandard.BSDTCPcode.
`Thetrackingvariablesalteredbyroutinesintcpinput.carelistedinTable.
`TrackingVariables-Receiving
`Name
`Description
`trcvackbyte
`bytesackedbytheacknowledgementpacketsreceived
`trcvackpack
`acknowledgementpacketsreceived
`trcvbyte
`bytesreceived
`trcvdupack
`duplicateacknowledgementpacketsreceived
`trcvdupbyte
`duplicatebytesreceived
`trcvpartdupbyteduplicatebytesinpartiallyduplicatepackets
`Table. tcpinput.cTrackingVariables
`Thetrackingvariablesthatareincrementedbyroutinesinthetcpinput.c(cid:12)le.
`..
`tcpoutput.c
`Allchangestotcpoutput.cinvolvetrackingthee(cid:11)ectsoftheextendedacknowl-
`edgementintervalsonTCP'ssendingside.Thetrackingvariablesthataremodi(cid:12)ed
`intcpoutput.carelistedin..
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 023
`
`

`

`
`TrackingVariables-Sending
`Name
`Description
`tsndacks
`ack-only(non-piggybacked)acknowledgementssent
`tsndbyte
`databytessent
`tsndrexmitbyte
`retransmittedbytes
`tsndrexmitpackretransmittedpackets
`Table.tcpoutput.cTrackingVariables
`Thetrackingvariablesthatareincrementedbyroutinesinthetcpoutput.c(cid:12)le.
`tcpsubr.c
`..
`WhenanewsocketiscreatedforaTCPconnection,thetcpnewtcpcbsubroutine
`iscalledtoallocateandinitializeanewTCPcontrolblock[WS ].Thecodethatwe
`addedtotcpnewtcpcballocatesandinitializesastructuretoholdtheextravariables
`neededforthisimplementation.TCPaccessesthesevariablesthroughapointerinside
`theTCPcontrolblockstructure.Thecontrolblockcannotholdtheinformationitself
`duetothesizerestrictionsoftheBSDmbufstructure[LMKQ ].
`AclosingTCPconnectioncallsthetcpclosefunction[WS ].Thecodeaddedto
`thisfunctionprintstheconnection'stracking(cid:12)eldvariablestothesystemlog.The
`tcpclosefunctionalsodeallocatesthememoryusedforthetracking(cid:12)eldsstructure.
`..
`tcptimer.c
`TherearetwotimerfunctionsforTCP.Thefasttimeriscalledeveryms,and
`theslowtimeriscalledeveryms.Thefasttimerfunctiondealswiththedelayed
`acknowledgementtimer[WS ].
`Thefasttimercheckstoseeifadelayedacknowledgementiswaitingtobesent.
`Itcheckseveryms,sothedelayedacknowledgementissentbetween and
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 024
`
`

`

`
`msfromthetimeitisprepared.Inadditiontosendingtheacknowledgement,the
`newimplementationresetstheoutstandingacknowledgementscount(tacknowcnt)
`tozero.Italsoincrementsthetrackingvariablethatcountsthenumberofdelayed
`acknowledgementstriggeredbythetimer(asopposedtothepresetoutstandingac-
`knowledgementlimit).
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 025
`
`

`

`
` .EXPERIMENTALRESULTS
`Weranthreegroupsofexperimentstotestthee(cid:11)ectofanextendedacknowledge-
`mentintervalonconnectionthroughput.The(cid:12)rstgroupoftestsranonthelocal
`network.Thesecondgroupoftestsranonanearnetwork,whichisseparatedfrom
`thelocalnetworkbyonebridge.Thethirdgroupoftestsranonadistantnetwork
`overtheInternet.Foreachtestrun,wetransferredalargeamountofdatafroma
`remotemachinetothemachinewiththealteredoperatingsystemandcapturedcon-
`nectionstatisticssuchasthroughputandnumberofretransmissionsforsubsequent
`analysis.
` ..StatisticalAnalysis
`Inordertomakeameaningfulcomparisonbetweenanytwotestbracketsusing
`con(cid:12)denceintervals,weneedanormaldistributionofdatapointsineachbracket
`[Fre ].However,wecannotassumethatasetoftestsmeasuringthroughputbetween
`twogivenmachineswillyieldanormaldistribution.Forexample,(cid:12)gure . contains
`agraphoftherawdataforanextendedacknowledgementintervalof .
`Thecentrallimittheoremstatesthat,givensetsofrandomsamplesfromanin(cid:12)-
`nitepopulation,themeansofthesetsarenormallydistributedregardlessoftheinitial
`distribution[Fre ].Tocomparethedatafromtwoextendedacknowledgementin-
`tervalsettings,wemust(cid:12)rstdividetestsfromeachacknowledgementintervalsetting
`intosets,andthentakethemeansofthesets.Themeanswillformanormaldistri-
`bution,whichcanbeusedtocomparethetwosettings.Figure .showstheadjusted
`throughputgraphfortheextendedacknowledgementinterval .Theadjustedcurve
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 026
`
`

`

`
`
`90
`
`80
`
`70
`
`Raw Local Net Throughput Distribution
`
`3.2
`
`4
`3.8
`3.6
`3.4
`Connection Throughput (MBit/s)
`
`4.2
`
`4.4
`
`4.6
`
`Figure . RawThroughputCurve
`Thisistherawthroughputdistributionfortheextendedacknowledgementinterval
`of ,withabu(cid:11)ersizeofk.
`isapproximatelynormal,whichwillallowcomparisonswiththeadjustedthroughput
`distributionfromanotheracknowledgementintervalsetting.
`Giventheresultsofthetests,wecansaywithacertaincon(cid:12)dencethatthetrue
`meanofthebracketpopulationlieswithinaspeci(cid:12)crange[Fre ].Forexample,
`giventheresultsofthelocaltests,wecansaywitha %con(cid:12)dencelevelthat
`themeanofthroughputsusinganextendedacknowledgementintervalofanda
`bu(cid:11)ersizeofkfallswithintherangeof.Mbit/secto.Mbit/sec.Ifthe
`con(cid:12)denceintervalsfortwoextendedacknowledgementintervalsoverlap,thenthere
`
`60
`
`50
`
`40
`
`30
`
`20
`
`10
`
`0
`2.8
`
`3
`
`Number of Connections
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 027
`
`

`

`
`
`Translated Local Net Throughput Distribution
`
`4.1
`
`4.16
`4.15
`4.14
`4.13
`4.12
`4.11
`Mean Connection Set Throughput
`
`4.17
`
`4.18
`
`0123456
`
`4.09
`
`Number of Connection Sets
`
`Figure .AdjustedThroughputCurve
`Thisistheadjustedthroughputdistributionfortheextendedacknowledgementin-
`tervalof ,withabu(cid:11)ersizeofk.
`isnostatisticaldi(cid:11)erencebetweenthem,i.e.wecannotclaimthatoneextended
`acknowledgementintervalgetsbetterresultsthantheother.
` . LocalNetwork
`Forthepurposesofthisresearch,twomachinesareconsideredtobeonthesame
`localnetworkiftheyareconnectedtothesamephysicalnetworkwithnobridgesor
`routersbetweenthem.Thelocalnetworktestsinvolvedthreemachines,whichareall
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 028
`
`

`

`
`connectedtothesamesectionofEthernet.ThereceivingTCPwasalwayslocatedon
`netipc,aSUNIPCworkstationrunningSunOS. .fromSunMicrosystems,with
`themodi(cid:12)cationslistedinchapter.ThesendingTCPwassometimesrunningon
`netipc ,amachineidenticaltonetipcbutrunninganunmodi(cid:12)edoperatingsystem.
`Theothersendingmachinewasjarok,aSUNSparcrunningSunOS. fromSun
`Microsystems.Experimentsranduringtheearlymorninghours,whenlocalnetwork
`tra(cid:14)cispresumedlow.
` . . Gatheringdata
`TomeasureTCPperformancebetweentwohosts,weusedaprogramcalledttcp.
`ItwascreatedattheU.S.ArmyBallisticsResearchLab(BRL)[Ste ].Thepro-
`gramproducesnetworktra(cid:14)candthenpresentsvariousperformancemeasurements,
`includingthroughput.
`Tomeasurethee(cid:11)ectofchangingtheextendedacknowledgementintervalonthe
`receivingmachine,weranseveraltestsatdi(cid:11)erentextendedacknowledgementinterval
`settings.Wecomparedtheresultsofthetestsusingthestatisticalmethodsoutlined
`insection ...
` . .ExperimentswithakWindowSize
`Asmentionedinsection . ,thebu(cid:11)ersizeatbothendsofaTCPconnection
`cane(cid:11)ectthroughputconsiderably.Thesizeofthesmallestbu(cid:11)eristhemaximum
`amountofdatathatmaybeintransitatagiventime.Weranthe(cid:12)rstsetof
`experimentsatTCP'sdefaultbu(cid:11)ersizesettingofk( )bytes.
`Overthousandconnectionsranbetweennetipc andnetipc.Thetestswere
`gatheredintosetsofforanalysis.Thegraphin(cid:12)gure . showsthe %con(cid:12)dence
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 029
`
`

`

`
`intervalscalculatedoverthenormallydistributedmeanofeachextendedacknowledge-
`mentinterval.Non-overlappingcon(cid:12)denceintervalsindicateastatisticallysigni(cid:12)cant
`di(cid:11)erenceinthroughput.Thegraphindicatesthatacknowledgingeverythirdseg-
`mentdoesimprovethroughput.However,thethroughputgainsdonotcontinueto
`riseforextendedacknowledgementintervalsoffourandabove.Thereisastatisti-
`callysigni(cid:12)cantdipinthethroughputatextendedacknowledgementinterval .This
`suggestssomeadverseinteractionwithinTCPatthisacknowledgementinterval.
`
`4.2
`
`4.15
`
`Local Net Throughput - 4k Buffer Size (95% conf)
`
`Mbps (avg and confidence interval)
`
`4.1
`
`4.05
`
`4
`
`3.95
`
`3.9
`
`2
`
`4
`
`6
`
`14
`12
`10
`8
`Acknowledgement Interval
`
`16
`
`18
`
`20
`
`Figure . LocalNetworkThroughput-kBu(cid:11)erSize
`Eachdatapointrepresentsthemeanofoneextendedacknowledgementinterval
`bracket.Non-overlappingcon(cid:12)denceintervalsindicateastatisticallysigni(cid:12)cantdif-
`ferenceinthroughput.
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 030
`
`

`

`
` . .. ExplanationofResults
`Inthelocalnetworkcase,extendedacknowledgementintervalsincreasedthrough-
`put.EachdatapointonFigure . representsoneextendedacknowledgementinterval
`setting.Thediamondrepresentsthethroughputmeanatthatinterval,andthever-
`ticalbarthroughthemeanextendsoverthecon(cid:12)denceintervalrange.Wecanstate
`with %con(cid:12)dencethatanextendedacknowledgmentintervalofthreeincreases
`throughput(fromthedefaultof)onalocalnetworkbasis.
`Inthelocalnetworkcase,theRTTdrawbacksofusinganextendedacknowledge-
`mentintervaldonotmakeasigni(cid:12)cantimpact.Oneoftheprincipaldrawbacksof
`usinganextendedacknowledgementintervalisthattheRTTfortheconnectionwill
`bein(cid:13)ated.Thisin(cid:13)atedRTTwillnotmatterunlessasegmentislostorcorrupted.
`Then,theconnectionwilltakelongerto(cid:12)ndoutaboutthesegmentloss.Onthe
`localnet,lostsegments(andthereforeretransmissions)arerare.Retransmittedseg-
`mentsmaybedetectedasbytesarrivingoutoforderintheconnection;throughout
`overtwenty-sixthousandlocalconnections,. percentofthedatabytesappeared
`outoforder.Thus,thein(cid:13)atedRTTseemstohaveaminimale(cid:11)ectonlocalnet
`connections.
`Havingfewerpacketsforagivenconnectionmayallowhigherthroughput[Cla].
`Aspreviouslystated,oneofthemajorsourcesofTCPoverheadisintheinterrupttime
`takentodealwitheachincomingpacket.Theextendedacknowledgementinterval
`allowsforfeweracknowledgementpacketsperconnection,whichcutsdownonthe
`overheadofprocessingpackets[Cla].Inaddition,moreoftheavailablenetwork
`resourcescanbeusedfordatasegments.Figure .showstheaveragenumberof
`acknowledgementsegmentssentfromthereceivingsideofthetestconnectionsat
`
`CAVIUM-1068
`Cavium, Inc. v. Alacritech, inc.
`Page 031
`
`

`

`
`eachacknowledgementintervaltested.Theamountofdatatobetransferredandthe
`windowsizewereconstantthroughouttheexperimentalruns.
`
`7000
`
`6000
`
`Avg Number of Acks per Connection - Local Net
`
`Average number of acks sent per connection
`
`5000
`
`4000
`
`3000
`
`200

This document is available on Docket Alarm but you must sign up to view it.


Or .

Accessing this document will incur an additional charge of $.

After purchase, you can access this document again without charge.

Accept $ Charge
throbber

Still Working On It

This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.

Give it another minute or two to complete, and then try the refresh button.

throbber

A few More Minutes ... Still Working

It can take up to 5 minutes for us to download a document if the court servers are running slowly.

Thank you for your continued patience.

This document could not be displayed.

We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.

Your account does not support viewing this document.

You need a Paid Account to view this document. Click here to change your account type.

Your account does not support viewing this document.

Set your membership status to view this document.

With a Docket Alarm membership, you'll get a whole lot more, including:

  • Up-to-date information for this case.
  • Email alerts whenever there is an update.
  • Full text search for other cases.
  • Get email alerts whenever a new case matches your search.

Become a Member

One Moment Please

The filing “” is large (MB) and is being downloaded.

Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.

Your document is on its way!

If you do not receive the document in five minutes, contact support at support@docketalarm.com.

Sealed Document

We are unable to display this document, it may be under a court ordered seal.

If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.


Access Government Site

We are redirecting you
to a mobile optimized page.





Document Unreadable or Corrupt

Refresh this Document
Go to the Docket

We are unable to display this document.

Refresh this Document
Go to the Docket