(12) United States Patent
`Hill et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,296,288 B1
`Nov. 13, 2007
`
`(54) METHODS, APPARATUSES, AND SYSTEMS
`ALLOWING FOR BANDWIDTH
`MANAGEMENT SCHEMES RESPONSIVE TO
`UTILIZATION CHARACTERISTICS
`ASSOCIATED WITH INDIVIDUAL USERS
`
`6,691,165 Bl* 2/2004 Bruck et al.
`6,934,745 B2 *
`8/2005 Krautkremer
`2003/0018889 Al *
`1/2003 Burnett et al.
`2003/0235209 Al * 12/2003 Garg et al.
`* cited by examiner
`
` 709/227
` 709/223
` 713/153
` 370/468
`
`(75)
`
`Inventors: Mark Hill, Los Gatos, CA (US); Guy
`Riddle, Los Gatos, CA (US); Robert E.
`Purvy, San Jose, CA (US)
`
`Primary Examiner Kim Vu
`Assistant Examiner Joseph Pan
`(74) Attorney, Agent, or Firm Mark J. Spolyar
`
`(73)
`
`Assignee: Packeteer, Inc., Cupertino, CA (US)
`
`(57)
`
`ABSTRACT
`
`* )
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 788 days.
`
`(21)
`
`Appl. No.: 10/295,391
`
`(22)
`
`Filed:
`
`Nov. 15, 2002
`
`(51)
`
`(52)
`(58)
`
`(2006.01)
`
`Int. Cl.
`G06F 21/00
` 726/2; 713/194
`U.S. Cl.
` 713/1,
`Field of Classification Search
`713/2, 188, 194, 193; 380/200, 201, 255,
`380/277; 726/2, 3, 11-15
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`Methods, apparatuses and systems allowing for bandwidth
`management schemes responsive to utilization characteris-
`tics associated with individual users. In one embodiment, the
`present invention allows network administrators to penalize
`users who carry out specific questionable or suspicious
`activities, such as the use of proxy tunnels to disguise the
`true nature of the data flows in order to evade classification
`and control by bandwidth management devices. In one
`embodiment, each individual user may be accorded an initial
`suspicion score. Each time the user is associated with a
`questionable or suspicious activity (for example, detecting
`the set up of a connection to an outside HTTP tunnel, or
`peer-to-peer application flow), his or her suspicion score is
`downgraded. Data flows corresponding to users with suffi-
`ciently low suspicion scores, in one embodiment, can be
`treated in a different manner from data flows associated with
`other users. For example, different or more rigorous classi-
`fication rules and policies can be applied to the data flows
`associated with suspicious users.
`
`6,339,784 Bl* 1/2002 Morris et al.
`6,484,203 Bl* 11/2002 Porras et al.
`
` 709/204
` 709/224
`
`34 Claims, 7 Drawing Sheets
`
`28
`
`sr.L
`
`26
`
`25
`
`50
`
`Computer
`Network
`
`24
`
`24
`
`21
`
`2 2 r'
`
`I
`
`42
`
`44
`
`II
`
`40
`
`(Outside)
`
`nside
`
`24
`
`30
`
`42
`
`42
`
`Cloudflare - Exhibit 1026, page 1
`
`Cloudflare - Exhibit 1026, page 1
`
`

`

`U.S. Patent
`
`Nov. 13, 2007
`
`Sheet 1 of 7
`
`US 7,296,288 B1
`
`28
`
`50
`
`Computer
`Network
`
`stir'
`ues
`
`Nc
`
`26
`
`25
`
`24
`
`21
`
`22
`
`42
`
`44
`
`40
`
`24
`
`24
`
`(Outside
`
`Inside)
`
`0 0 0
`
`30
`
`Fig._1
`
`42
`
`42
`
`Cloudflare - Exhibit 1026, page 2
`
`Cloudflare - Exhibit 1026, page 2
`
`

`

`U.S. Patent
`
`Nov. 13, 2007
`
`Sheet 2 of 7
`
`US 7,296,288 B1
`
`
`p50
`Administrator
`Interface
`I
`
` 140
`I
`
`F.)Measurement
`Engine
`
`138
`
`Suspicion
`Scoring Module
`
`Flow Control
`Module
`
`Data Packet
`Out
`
`137
`
`i
`Traffic
`Classification
`Database
`
`134
`
`Flow
`Database
`
`Host
`Database
`
`Data Packet
`In
`
`Packet
`Processor
`
`132
`
`Fig._2
`
`Cloudflare - Exhibit 1026, page 3
`
`Cloudflare - Exhibit 1026, page 3
`
`

`

`U.S. Patent
`
`Nov. 13, 2007
`
`Sheet 3 of 7
`
`US 7,296,288 B1
`
` ji Construct
`
`Control Block
`
`212
`
`I.<Receive Data
`
`Packet
`
`202
`
`204
`
`Yes
`
`208
`No
`
`New Data
`Flow?
`
`Control
`Block?
`
`Ye
`
`218
`
`K_
`
`Fetch/Update
`Control Block
`
`220
`
`N
`
`Changes
`To Flow?
`
`Yes
`
`Identify
`Traffic Class
`
`214
`
`216
`
`222
`
`224
`
`---
`
`11.1g . —j
`
`P = getControls
`(Traffic Class)
`
`Pass Packet to
`Flow Control
`Module (P)
`
`J
`
`V
`Record Bandwidth
`Utilization Data In
`Association with
`Traffic Class
`
`Cloudflare - Exhibit 1026, page 4
`
`Cloudflare - Exhibit 1026, page 4
`
`

`

`U.S. Patent
`
`Nov. 13, 2007
`
`Sheet 4 of 7
`
`US 7,296,288 B1
`
`AccessLink
`
`Inbound
`
`LocalHost
`
`SuspiciousUsers
`
`HTTP
`
`I Telnet
`FTP
`
`ell Default
`
`Outbound
`
`LocalHost
`
`141 SuspiciousUsers
`
`rat H111)
`
`Telnet
`FTP
`
`Default
`
`Fig. 4A
`
`AccessLink
`
`Inbound
`
`
`
`
`
`, LocalHost
`
` SuspiciousUsers
`
`
`
`
`
`IPAddr1
`
`1PAddr2
`FPAddr3
`
`HTTP
`
`Telnet
`FTP
`
`Default
`
`Outbound
`
`' LocalHost
` SuspiciousUsers
`
`
`
`IPAddr1
`
`IPAddr2
`IPAddr3
`
`
`
`
`
`
`
`HI1P
`
`Telnet
`FTP
`
`Default
`
`Cloudflare - Exhibit 1026, page 5
`
`Cloudflare - Exhibit 1026, page 5
`
`

`

`U.S. Patent
`
`Nov. 13, 2007
`
`Sheet 5 of 7
`
`US 7,296,288 B1
`
`302
`
`304
`
`New Data
`Flow?
`
`Yes
`
`310
`
`New
`User?
`
`Yes
`)0-
`
`Instantiate
`Suspicion Scoring
`Object
`
`Pickled
`Object?
`
`306
`
`Yes
`
`308
`
`Un-Pickle
`Suspicion Scoring
`Object
`
`312
`
`d
`
`Pass Packet to
`Suspicion Scoring <
`Object
`
`Fig. _5
`
`Cloudflare - Exhibit 1026, page 6
`
`Cloudflare - Exhibit 1026, page 6
`
`

`

`U.S. Patent
`
`Nov. 13, 2007
`
`Sheet 6 of 7
`
`US 7,296,288 B1
`
`Client Device
`
`71
`
`72
`
`-----.._.
`
`PtoP App
`
`Tunnel
`Client
`
`42
`
`30
`
`0 0 0
`
`50
`
`74
`
`Computer
`Network
`
`Tunnel Proxy
`Server
`
`50
`
`Computer
`Network
`
`75
`
`Network
`Resource
`
`Fig._6
`
`Cloudflare - Exhibit 1026, page 7
`
`Cloudflare - Exhibit 1026, page 7
`
`

`

`U.S. Patent
`
`Nov. 13, 2007
`
`Sheet 7 of 7
`
`US 7,296,288 B1
`
`IP Address
`216.203.49.219
`216.148.237.158
`216.148.237.145
`10.7.15.8
`10.255.255.255
`10.7.15.4
`10.1.1.40
`207.46.249.61
`10.1.1.16
`255.255.255.255
`10.7.11.2
`'10.7.15.13
`66.218.71.83
`10.2.1.10
`239.255.255.253
`10.7.15.5
`10.1.1.18
`:10.10.254.74
`10.10.253.70
`10.7.31.22
`
`Conn RTT to PS Curr Rate 1 Min Aug Peak Rate
`2730
`420
`'
`0
`80ms
`2730
`19.1k
`235k
`14ms
`235k
`1
`49k
`4871
`48ms
`2303
`5
`58.9k
`310k
`310k
`11
`3ms
`17.6k
`3464
`3397
`2
`1112
`654
`190
`0
`1112
`643
`188
`0
`220k
`15.2k
`20.5k
`0
`17.7k
`2905
`3657
`1
`5357
`1735
`430
`1
`11.1k
`0
`0
`0
`11.3k
`252
`549
`2
`90.6k
`4217
`492
`2
`343
`0
`0
`0
`1305
`25
`2
`39
`8787
`318
`485
`0
`2091
`771
`1
`1349
`54
`345
`0
`0
`37
`0
`0
`0
`0
`2
`0
`0
`
`***
`
`.**
`
`***
`
`25ms
`***
`
`***
`
`124ms
`***
`
`***
`
`***New Flows Per Minute***
`Failed
`Client
`Server
`0
`90
`0
`84
`0
`0
`64
`0
`0
`0
`42
`0
`0
`28
`0
`0
`11
`6
`6
`11
`0
`0
`14
`0
`0
`0
`11
`4
`0
`0
`0
`0
`3
`0
`0
`3
`2
`0
`0
`0
`0
`1
`1
`0
`0
`0
`1
`0
`0
`0
`1
`0
`0
`1
`0
`0
`0
`0
`0
`0
`
`l
`
`Table 7
`
`Cloudflare - Exhibit 1026, page 8
`
`Cloudflare - Exhibit 1026, page 8
`
`

`

`1
`METHODS, APPARATUSES, AND SYSTEMS
`ALLOWING FOR BANDWIDTH
`MANAGEMENT SCHEMES RESPONSIVE TO
`UTILIZATION CHARACTERISTICS
`ASSOCIATED WITH INDIVIDUAL USERS
`
`COPYRIGHT NOTICE
`
`A portion of the disclosure of this patent document
`contains material which is subject to copyright protection.
`The copyright owner has no objection to the facsimile
`reproduction by anyone of the patent document or the patent
`disclosure as it appears in the Patent and Trademark Office
`patent file or records, but otherwise reserves all copyright
`rights whatsoever.
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application makes reference to the following com-
`monly owned U.S. patent applications and patents, which
`are incorporated herein by reference in their entirety for all
`purposes:
`U.S. patent application Ser. No. 08/762,828 now U.S. Pat.
`No. 5,802,106 in the name of Robert L. Packer, entitled
`"Method for Rapid Data Rate Detection in a Packet Com-
`munication Environment Without Data Rate Supervision;"
`U.S. patent application Ser. No. 08/970,693 now U.S. Pat.
`No. 6,018,516, in the name of Robert L. Packer, entitled
`"Method for Minimizing Unneeded Retransmission of Pack-
`ets in a Packet Communication Environment Supporting a
`Plurality of Data Link Rates;"
`U.S. patent application Ser. No. 08/742,994 now U.S. Pat.
`No. 6,038,216, in the name of Robert L. Packer, entitled
`"Method for Explicit Data Rate Control in a Packet Com-
`munication Environment without Data Rate Supervision;"
`U.S. patent application Ser. No. 09/977,642 now U.S. Pat.
`No. 6,046,980, in the name of Robert L. Packer, entitled
`"System for Managing Flow Bandwidth Utilization at Net-
`work, Transport and Application Layers in Store and For-
`ward Network;"
`U.S. patent application Ser. No. 09/106,924 now U.S. Pat.
`No. 6,115,357, in the name of Robert L. Packer and Brett D.
`Galloway, entitled "Method for Pacing Data Flow in a
`Packet-based Network;"
`U.S. patent application Ser. No. 09/046,776 now U.S. Pat.
`No. 6,205,120, in the name of Robert L. Packer and Guy
`Riddle, entitled "Method for Transparently Determining and
`Setting an Optimal Minimum Required TCP Window Size;"
`U.S. patent application Ser. No. 09/479,356 now U.S. Pat.
`No. 6,285,658, in the name of Robert L. Packer, entitled
`"System for Managing Flow Bandwidth Utilization at Net-
`work, Transport and Application Layers in Store and For-
`ward Network;"
`U.S. patent application Ser. No. 09/198,090 now U.S. Pat.
`No. 6,412,000, in the name of Guy Riddle and Robert L.
`Packer, entitled "Method for Automatically Classifying
`Traffic in a Packet Communications Network;"
`U.S. patent application Ser. No. 09/198,051, in the name
`of Guy Riddle, entitled "Method for Automatically Deter-
`mining a Traffic Policy in a Packet Communications Net-
`work;"
`U.S. patent application Ser. No. 09/206,772, in the name
`of Robert L. Packer, Brett D. Galloway and Ted Thi, entitled
`"Method for Data Rate Control for Heterogeneous or Peer
`Internetworking;"
`
`US 7,296,288 B1
`
`5
`
`2
`U.S. patent application Ser. No. 09/885,750, in the name
`of Scott Hankins and Brett Galloway, entitled "System and
`Method For Dynamically Controlling a Rogue Application
`Through Incremental Bandwidth Restrictions;"
`U.S. patent application Ser. No. 09/966,538, in the name
`of Guy Riddle, entitled "Dynamic Partitioning of Network
`Resources;"
`U.S. patent application Ser. No. 10/039,992, in the
`Michael J. Quinn and Mary L. Laier, entitled "Method and
`10 Apparatus for Fast Lookup of Related Classification Entities
`in a Tree-Ordered Classification Hierarchy;"
`U.S. patent application Ser. No. 10/015,826, in the name
`of Guy Riddle, entitled "Dynamic Tunnel Probing in a
`Communications Network;"
`15 U.S. patent application Ser. No. 10/108,085, in the name
`of Wei-Lung Lai, Jon Eric Okholm, and Michael J. Quinn,
`entitled "Output Scheduling Data Structure Facilitating
`Hierarchical Network Resource Allocation Scheme;"
`U.S. patent application Ser. No. 10/155,936, in the name
`20 of Guy Riddle, Robert L. Packer and Mark Hill, entitled
`"Method
`for Automatically Classifying Traffic with
`Enhanced Hierarchy in a Packet Communications Net-
`work;"
`25 U.S. patent application Ser. No. 10/177,518, in the name
`of Guy Riddle, entitled "Methods, Apparatuses and Systems
`Allowing for Progressive Network Resource Utilization
`Control Scheme;" and
`U.S. patent application Ser. No. 10/178,617, in the name
`30 of Robert E. Purvy, entitled "Methods, Apparatuses and
`Systems Facilitating Analysis of Network Device Perfor-
`mance."
`
`FIELD OF THE INVENTION
`
`35
`
`40
`
`The present invention relates to computer networks and
`bandwidth management, and, more particularly, to methods,
`apparatuses and systems allowing for bandwidth manage-
`ment schemes responsive to the utilization characteristics
`associated with individual users.
`
`BACKGROUND OF THE INVENTION
`
`In order to understand the context of certain embodiments
`4.s of the invention, the following provides an explanation of
`certain technical aspects of a packet based telecommunica-
`tions network environment. Internet/Intranet technology is
`based largely on the TCP/IP protocol suite. At the network
`level, IP provides a "datagram" delivery service
`that is, IP
`50 is a protocol allowing for delivery of a datagram or packet
`between two hosts. By contrast, TCP provides a transport
`level service on top of the datagram service allowing for
`guaranteed delivery of a byte stream between two IP hosts.
`In other words, TCP is responsible for ensuring at the
`55 transmitting host that message data is divided into packets to
`be sent, and for reassembling, at the receiving host, the
`packets back into the complete message.
`TCP has "flow control" mechanisms operative at the end
`stations only to limit the rate at which a TCP endpoint will
`60 emit data, but it does not employ explicit data rate control.
`The basic flow control mechanism is a "sliding window", a
`window which by its sliding operation essentially limits the
`amount of unacknowledged transmit data that a transmitter
`is allowed to emit. Another flow control mechanism is a
`65 congestion window, which is a refinement of the sliding
`window scheme involving a conservative expansion to make
`use of the full, allowable window.
`Cloudflare - Exhibit 1026, page 9
`
`Cloudflare - Exhibit 1026, page 9
`
`

`

`US 7,296,288 B1
`
`3
`The sliding window flow control mechanism works in
`conjunction with the Retransmit Timeout Mechanism
`(RTO), which is a timeout to prompt a retransmission of
`unacknowledged data. The timeout length is based on a
`running average of the Round Trip Time (RTT) for acknowl-
`edgment receipt, i.e. if an acknowledgment is not received
`within (typically) the smoothed RTT+4*mean deviation,
`then packet loss is inferred and the data pending acknowl-
`edgment is re-transmitted. Data rate flow control mecha-
`nisms which are operative end-to-end without explicit data
`rate control draw a strong inference of congestion from
`packet loss (inferred, typically, by RTO). TCP end systems,
`for example, will "back-off,"
`i.e., inhibit transmission in
`increasing multiples of the base RTT average as a reaction
`to consecutive packet loss.
`A crude form of bandwidth management in TCP/IP net-
`works (that is, policies operable to allocate available band-
`width from a single logical link to network flows) is accom-
`plished by a combination of TCP end systems and routers
`which queue packets and discard packets when some con-
`gestion threshold is exceeded. The discarded and therefore
`unacknowledged packet serves as a feedback mechanism to
`the TCP transmitter. Routers support various queuing
`options to provide for some level of bandwidth manage-
`ment. These options generally provide a rough ability to
`partition and prioritize separate classes of traffic. However,
`configuring these queuing options with any precision or
`without side effects is in fact very difficult, and in some
`cases, not possible. Seemingly simple things, such as the
`length of the queue, have a profound effect on traffic
`characteristics. Discarding packets as a feedback mechanism
`to TCP end systems may cause large, uneven delays per-
`ceptible to interactive users. Moreover, while routers can
`slow down inbound network traffic by dropping packets as
`a feedback mechanism to a TCP transmitter, this method
`often results in retransmission of data packets, wasting
`network traffic and, especially, inbound capacity of a WAN
`link. In addition, routers can only explicitly control out-
`bound traffic and cannot prevent inbound traffic from over-
`utilizing a WAN link. A 5% load or less on outbound traffic
`can correspond to a 100% load on inbound traffic, due to the
`typical imbalance between an outbound stream of acknowl-
`edgments and an inbound stream of data.
`In response, certain data flow rate control mechanisms
`have been developed to provide a means to control and
`optimize efficiency of data transfer as well as allocate
`available bandwidth among a variety of business enterprise
`functionalities. For example, U.S. Pat. No. 6,038,216 dis-
`closes a method for explicit data rate control in a packet-
`based network environment without data rate supervision.
`Data rate control directly moderates the rate of data trans-
`mission from a sending host, resulting in just-in-time data
`transmission to control inbound traffic and reduce the inef-
`ficiencies associated with dropped packets. Bandwidth man-
`agement devices allow for explicit data rate control for flows
`associated with a particular traffic classification. For
`example, U.S. Pat. No. 6,412,000, above, discloses auto-
`matic classification of network traffic for use in connection
`with bandwidth allocation mechanisms. U.S. Pat. No. 6,046,
`980 discloses systems and methods allowing for application
`layer control of bandwidth utilization in packet-based com-
`puter networks. For example, bandwidth management
`devices allow network administrators to specify policies
`operative to control and/or prioritize the bandwidth allocated
`to individual data flows according to traffic classifications. In
`addition, certain bandwidth management devices, as well as
`certain routers, allow network administrators to specify
`
`4
`aggregate bandwidth utilization controls to divide available
`bandwidth into partitions. With some network devices, these
`partitions can be configured to ensure a minimum bandwidth
`and/or cap bandwidth as to a particular class of traffic. An
`5 administrator specifies a traffic class (such as FTP data, or
`data flows involving a specific user) and the size of the
`reserved virtual link
`i.e., minimum guaranteed bandwidth
`and/or maximum bandwidth. Such partitions can be applied
`on a per-application basis (protecting and/or capping band-
`10 width for all traffic associated with an application) or a
`per-user basis (controlling, prioritizing, protecting and/or
`capping bandwidth for a particular user). In addition, certain
`bandwidth management devices allow administrators to
`define a partition hierarchy by configuring one or more
`15 partitions dividing the access link and further dividing the
`parent partitions into one or more child partitions.
`While the systems and methods discussed above that
`allow for traffic classification and application of bandwidth
`utilization controls on a per-traffic-classification basis oper-
`2o ate effectively for their intended purposes, they possess
`certain limitations. As discussed more fully below, identifi-
`cation of traffic types associated with data flows traversing
`an access link involves the application of matching criteria
`or rules to various characteristics of the data flows. Such
`25 matching criteria can include source and destination IP
`addresses, port numbers, MIME types, etc. After identifica-
`tion of a traffic type corresponding to a data flow, a band-
`width management device associates and subsequently
`applies bandwidth utilization controls (e.g., a policy or
`30 partition) to the data flow corresponding to the identified
`traffic classification or type. A common use of bandwidth
`management devices is to limit the bandwidth being con-
`sumed by unruly, bandwidth-intensive applications, such as
`peer-to-peer applications (e.g., Kazaa, Napster, etc.). Net-
`35 work savvy users (such as students in a campus or university
`environment), however, have become aware that such band-
`width management devices have been deployed to limit or
`restrict such unauthorized network traffic. As a result, users
`often attempt to bypass or thwart the bandwidth manage-
`40 ment scheme effected by such bandwidth management
`devices by creating communications tunnels (proxy tunnels)
`through which unauthorized or restricted network traffic is
`sent. The attributes discernible from the content of these
`tunneled data flows, however, often reveal little information
`45 about its true nature. For example, commercial HTTP tunnel
`services (such as loopholesoftware.com, TotalRc.net, and
`http-tunnel.com, etc.) allow users to send all network traffic
`in the form of HTTP traffic through a HTTP tunnel between
`a tunnel client and an HTTP proxy server maintained by the
`so tunnel services provider. FIG. 6 illustrates the functionality
`and operation of a typical HTTP proxy tunnel. Client device
`42 includes a client application (such as a peer-to-peer
`application 71) and a tunnel client 72. The client application
`sends data to the tunnel client 72 which tunnels the data over
`55 HTTP to a tunnel proxy server 74. The tunnel proxy server
`74 then forwards the data to the intended destination (here,
`network resource 75), and vice versa. Such HTTP tunnels
`typically feature encryption; accordingly, a bandwidth man-
`agement device 30, encountering the tunneled traffic in this
`60 form, may not detect the exact nature of the traffic and, in
`fact, classify such data flows as legitimate or regular HTTP
`traffic. Accordingly, these tunneling mechanisms and other
`techniques for evading bandwidth utilization controls imple-
`mented by bandwidth management devices present new
`65 challenges to network administrators and bandwidth device
`manufacturers desiring to effectively control unauthorized
`or restricted network traffic.
`Cloudflare - Exhibit 1026, page 10
`
`Cloudflare - Exhibit 1026, page 10
`
`

`

`US 7,296,288 B1
`
`6
`DESCRIPTION OF PREFERRED
`EMBODIMENT(S)
`
`I. Exemplary Operating Environment
`
`5
`In light of the foregoing, a need in the art exists for
`methods, apparatuses and systems allowing for bandwidth
`management schemes that are responsive to the utilization
`characteristics associated with individual users. A need in
`the art further exists for methods, apparatuses and systems 5
`allowing for detection of questionable or other activities
`designed to evade bandwidth management control schemes
`and, thus, enabling application of more rigorous network
`traffic classification mechanisms and/or disparate bandwidth
`utilization controls. Embodiments of the present invention
`substantially fulfill these needs.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides methods, apparatuses and
`systems allowing for bandwidth management schemes
`responsive to utilization characteristics associated with indi-
`vidual users. In one embodiment, the present invention
`allows network administrators to penalize users who carry
`out specific questionable or suspicious activities, such as the
`use of proxy tunnels to disguise the true nature of the data
`flows in order to evade classification and control by band-
`width management devices. In one embodiment, each indi-
`vidual user may be accorded an initial suspicion level. Each
`time the user is associated with a questionable or suspicious
`activity (for example, detecting the setup of a connection to
`an outside HTTP tunnel, or peer-to-peer application flow),
`his or her suspicion level is adjusted. Data flows correspond-
`ing to users with sufficiently high suspicion levels, in one
`embodiment, can be treated in a different manner from data
`flows associated with other users. For example, different or
`more rigorous classification rules and bandwidth manage-
`ment policies can be applied to the data flows associated
`with suspicious users. For example, data flows associated
`with suspicious users may be examined more closely in
`order to determine more thoroughly or accurately appropri-
`ate classification rules and/or bandwidth management poli-
`cies.
`
`DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a functional block diagram illustrating a com-
`puter network environment including a bandwidth manage-
`ment device according to an embodiment of the present
`invention.
`FIG. 2 is a functional block diagram setting forth the
`functionality in a bandwidth management device according
`to an embodiment of the present invention.
`FIG. 3 is a flow chart providing a method directed to
`processing data packets to allow for enforcement of band-
`width utilization and other controls on network data flows.
`FIG. 4A is a diagram illustrating a traffic classification
`configuration for a given access link according to an
`embodiment of the present invention.
`FIG. 4B is a diagram illustrating a traffic classification
`configuration for a given access link according to another
`embodiment of the present invention.
`FIG. 5 is a flow chart diagram setting forth a method
`directed to the management of suspicion scoring objects
`according to an embodiment of the present invention.
`FIG. 6 is a functional block diagram illustrating a proxy
`tunnel which may be used in attempts to circumvent the
`bandwidth utilization controls implemented by bandwidth
`management devices.
`Table 7 sets forth the data flow metrics, according to an
`embodiment of the present invention, maintained for each
`host associated with data flows traversing a bandwidth
`management device.
`
`FIG. 1 sets forth a packet-based computer network envi-
`ronment including a bandwidth management device 30. As
`FIG. 1 shows, local area computer network 40 interconnects
`several TCP/IP end systems, including client devices 42 and
`10 server device 44, and provides access to resources operably
`connected to computer network 50 via router 22 and access
`link 21. Access link 21 is a physical and/or logical connec-
`tion between two networks, such as computer network 50
`and local area network 40. Server 28 is a TCP end system
`15 connected to computer network 50 through router 26 and
`access link 25. Client devices 24 are additional TCP end
`systems operably connected to computer network 50 by any
`suitable means, such as through an Internet Services Pro-
`vider (ISP). The computer network environment, including
`20 computer network 50 is a packet-based communications
`environment, employing TCP/IP protocols, and/or other
`suitable protocols, and has a plurality of interconnected
`digital packet transmission stations or routing nodes. Band-
`width management device 30 is provided between router 22
`25 and local area computer network 40. Bandwidth manage-
`ment device 30 is operative to classify data flows and,
`depending on the classification, enforce respective band-
`width utilization controls on the data flows to control
`bandwidth utilization and optimize network application per-
`30 formance across access link 21.
`
`A. Bandwidth Management Device
`FIG. 2 is a block diagram illustrating functionality,
`according to one embodiment of the present invention,
`35 included in bandwidth management device 30. In one
`embodiment, bandwidth management device 30 comprises
`packet processor 131, flow control module 132, measure-
`ment engine 140, traffic classification engine 137, suspicion
`scoring module 138, and administrator interface 150. Packet
`40 processor 131 is operative to detect new data flows and
`construct data structures including attributes characterizing
`the data flow. Flow control module 132 is operative to
`enforce bandwidth utilization controls on data flows travers-
`ing bandwidth management device 30. Traffic classification
`45 engine 137 is operative to analyze data flow attributes and
`identify traffic classes corresponding to the data flows, as
`discussed more fully below. In one embodiment, traffic
`classification engine 137 stores traffic classes associated
`with data flows encountered during operation of bandwidth
`so management device 30, as well as manually created traffic
`classes and a hierarchical traffic class structure, if any,
`configured by a network administrator. In one embodiment,
`traffic classification engine 137 stores traffic classes, in
`association with pointers to bandwidth utilization controls or
`55 pointers to data structures defining such bandwidth utiliza-
`tion controls. Suspicion scoring module 138 is operative to
`examine data flows associated with individual users and
`evaluate whether characteristics of the data flows indicate
`suspicious activity (e.g., an attempt to evade classification
`60 and, therefore, configured bandwidth management controls,
`and/or indications that such attempts may be likely). Mea-
`surement engine 140 maintains measurement data relating to
`operation of bandwidth management device 30 to allow for
`monitoring of bandwidth utilization across access link 21
`65 with respect to a plurality of bandwidth utilization and other
`network statistics on an aggregate and/or per-traffic-class
`level.
`Cloudflare - Exhibit 1026, page 11
`
`Cloudflare - Exhibit 1026, page 11
`
`

`

`US 7,296,288 B1
`
`7
`Administrator interface 150 facilitates the configuration
`of bandwidth management device 30 to adjust or change
`operational and configuration parameters associated with the
`device. For example, administrator interface 150 allows
`administrators to select identified traffic classes and associ-
`ate them with bandwidth utilization controls, such as a
`partition, as well as other controls. Administrator interface
`150 also displays various views associated with a hierarchi-
`cal traffic classification scheme and allows administrators to
`configure or revise the hierarchical traffic classification
`scheme as discussed more fully below. Administrator inter-
`face 150 can be a command line interface or a graphical user
`interface accessible, for example, through a conventional
`browser on client device 42.
`A.1. Packet Processing
`In one embodiment, when packet processor 131 encoun-
`ters a new data flow it stores the source and destination IP
`addresses contained in the packet headers in host database
`134. Packet processor 131 further constructs a control block
`object including attributes characterizing a specific flow
`between two end systems. In one embodiment, a control
`block object contains a flow specification object (or a pointer
`thereto) including such attributes as pointers to the "inside"
`and "outside" IP addresses in host database 134, as well as
`other flow specification parameters, such as inside and
`outside port numbers, service type, protocol type and other
`parameters characterizing the data flow. In one embodiment,
`such parameters can include information gleaned from
`examination of data within layers 2 through 7 of the OSI
`reference model. U.S. Pat. No. 6,046,980, incorporated by
`reference herein, discloses classification of data flows for
`use in a packet-based communications environment. FIG. 1
`illustrates the concept associated with inside and outside
`addresses. As discussed above, in one embodiment, a flow
`specification object includes an "inside" and "outside"
`address relative to bandwidth management device 30. See
`FIG. 1. For a TCP/IP packet, packet processor 131 can
`compute the inside and outside addresses based on the
`source and destination addresses of the packet and the
`direction of the packet flow.
`In one embodiment, packet processor 131 creates and
`stores control block objects corresponding to data flows in
`flow database 135. In one embodiment, control block object
`attributes include a pointer to a corresponding flow speci-
`fication object, as well as other flow state parameters, such
`as TCP connection status, timing of last packets in the
`inbound and outbound directions, speed information, appar-
`ent round trip time, number of packets, aggregate bytes, etc.
`Control block object attributes further include at least one
`traffic class identifier (or pointer(s) thereto) associated with
`the data flow, as welt as policy parameters (or pointers
`thereto) corresponding to the identified traffic class. In one
`embodiment, control block objects further include a list of
`traffic classes for which measurement data associated with
`the data flow should be logged. In one embodiment, to
`facilitate association of an existing control block object to
`subsequent packets associated with a data flow or connec-
`tion, flow database 135 further maintains a control block
`hash table including a key comprising a hashed value
`computed from a string comprising the inside IP address,
`outside IP address, inside port number, outside port number,
`and protocol type (e.g., TCP, UDP, etc.) associated with a
`pointer to the corresponding control block object. According
`to this embodiment, to identify whether a control block
`object exists for a given data flow, packet processor 131
`hashes the values identified above and scans the hash table
`
`8
`for a matching entry. If one exists, packet processor 131
`associates the pointer to the corresponding control block
`object with the data flow.
`As discussed above, host database 134 stores the IP or
`5 other computer network addresses associated with the end
`systems identified in data flows traversing bandwidth man-
`agement device. In one embodiment, host database 134
`maintains for each IP address, an "inside/outside" flag value
`set to indicate whether the IP address corresponds to an
`10 "inside" or "outside" system (see above), and a suspicion
`level, score or other value computed by suspicion scoring
`module 138, as described more fully below. In other embodi-
`ments, host database 134 can also be extended to maintain
`other suspicion score values, such as an average suspicion
`15 score over an interval, the lowest/highest suspicion score
`over an interval, etc. In one embod