`
`111111111111 11111111111111JII,111111111111111111111111111111
`
`(12) United States Patent
`Turner et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,313,100 B1
`Dec. 25, 2007
`
`(54)
`
`NETWORK DEVICE HAVING ACCOUNTING
`SERVICE CARD
`
`1/2001 Graham-Ctumning,
`6,182,146 B1
`6,321,338 Bl* 11/2001 Porras et al.
`
`Jr.
` 726/25
`
`(75)
`
`Inventors: Stephen W Turner, Menlo Park, CA
`(US); Hsien-Chung Woo, Fremont, CA
`(US); Sanjay KaIra, San Jose, CA
`(US); Truman Joe, Mountain View, CA
`(US); Wendy R Cartee, Los Altos, CA
`(US)
`
`(73)
`
`Assignee: Juniper Networks, Inc., Sunnyvale,
`CA (US)
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`WO
`WO
`
`WO 9836532 Al * 8/1998
`WO 2084920 A2 * 10/2002
`
`OTHER PUBLICATIONS
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1034 days.
`
`Weaver, A.C. et al., "A Real-Time Monitor for Token Ring Net-
`works," Military Communications Conference, 1989. MILCOM
`'89. Oct. 1989. vol. 3. pp. 794-798.*
`
`(21)
`
`Appl. No.: 10/228,150
`
`(22)
`
`Filed:
`
`Aug. 26, 2002
`
`(51)
`
`Int. Cl.
`HO4L 12/26
`
`(52)
`
`U.S. Cl.
`
`(2006.01)
`
` 370/253; 370/244; 370/252;
`370/392
`
`(58)
`
`Field of Classification Search
` 370/235,
`370/242-244, 250, 252, 253, 389, 392, 396,
`370/469, 471; 709/223, 224, 229
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6/1976 Requa et al.
`3,962,681 A
`6/1977 Jenny et al.
`4,032,899 A
`7/1986 Everett, Jr.
`4,600,319 A
`4/1995 Finlay et al.
`5,408,539 A
`5,490,252 A * 2/1996 Macera et al.
`5,509,123 A
`4/1996 Dobbins et al.
`5,568,471 A * 10/1996 Hershey et al.
`6,011,795 A
`1/2000 Varghese et al.
`6.018,765 A
`1/2000 Durana et al.
`6,148,335 A * 11/2000 Haggard et al.
`
`709/249
`
`370/245
`
`709/224
`
`(Continued)
`
`Primary Examiner—Chi Pham
`Assistant Examiner—Donald L Mills
`(74) Attorney, Agent, or Finn—Shumaker & Sieffert P.A.
`
`(57)
`
`ABSTRACT
`
`A network device integrates accounting functionality for
`generation of flow statistics with packet intercept function-
`ality to provide a comprehensive traffic analysis environ-
`ment. The device comprises a set of network interface cards
`to receive packets from a network, and a set of accounting
`service cards to calculate flow statistics for the packets. The
`device further comprises a control unit to receive the net-
`work packets from the interface cards and distribute the
`packets to the set of accounting service cards. The account-
`ing service card comprises an interface for insertion within
`a slot of a network device. Accounting service cards may be
`added to easily scale the network device to support higher
`bandwidth communication links, such as OC-3, OC-12,
`00048 and higher rate links. Additional accounting service
`cards may be used for purposes of redundancy to support
`continuous, uninterrupted packet processing and accounting
`in the event of a card failure.
`
`24 Claims, 9 Drawing Sheets
`
`ACCOUNThO
`SC
`
`PACKET PACKET
`STREAM STREAM
`A
`
`I
`
`I
`
`A
`
`TUNNEL
`sc
`
`CONTRCI.
`UNIT
`St
`
`tn.
`4T-64
`L. 4.
`
`ENCRYPTION
`SC
`
`14
`
`SAMPLED
`FLOW
`RECORDS TRAFFIC
`
`Cloudflare - Exhibit 1023, page 1
`
`Cloudflare - Exhibit 1023, page 1
`
`

`

`US 7,313,100 Bl
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5/2002 Hjalmtysson
`6,392,996 B1
`12/2002 Wexler et al.
`6,499,088 B1
`6,563,796 31 * 5/2003 Saito
`7/2003 Uzun
`6,590,898 B1
`7/2003 Aukia et al.
`6,594,268 B1
`7/2003 Kloth
`6,598,034 B1
`5/2004 Mahajan et al.
`6,735,201 B1
`6/2004 Farrell et al.
`6,751,663 81
`11/2004 Beesley et al.
`6,826,713 B1
`6,983,294 B2 * 1/2006 Jones et al.
`6,985,956 B2 * 1/2006 Luke et al.
`Jungck et al.
`9/2006
`7,114,008 B2
`2002/0141343 Al
`10/2002
`Bays
`1/2003 Bullard
`2003/0005145 Al
`6/2003 McCollom et al.
`2003/0120769 Al
`11/2003 Kan et al.
`2003/0214913 Al
`
`370/252
`
`707/202
`709/229
`
`OTHER PUBLICATIONS
`Dini, P. et al., "Performance Evaluation for Distributed System
`Components," Proceedings of IEEE Second International Workshop
`on Systems Management. Jun. 1996. pp. 20-29.*
`Integrated Services Adapter, 2000, Cisco Systems, Data Sheet, pp.
`http://www.cisco.com/warp/public/cc/pd/ifaa/svaa/iasvaa/
`1-6,
`prodlit/ism2_ds.pdf.
`"The CAIDA Web Site," wwwcaida.org/, 2000.
`"About Endace," www.endace.com/, 2000.
`"Cisco IOS NetFlow," www.cisco.com/warp/public/732/Tech/nmp/
`netflow/index.shtml, 2002.
`U.S. Appl. No. 10/188,567, entitled "Adaptive Network Flow
`Analysis", filed Jul. 2, 2002, Scott Mackie.
`U.S. Appl. No. 10/228,132, entitled "Adaptive Network Router",
`filed Aug. 26, 2002, Woo et al.
`U.S. Appl. No. 10/228,114, entitled "Network Router Having Inte-
`grated Flow Accounting and Packet Interception", filed Aug. 26,
`2002, Woo et al.
`U.S. Appl. No. 10/241,785, entitled "Rate-Controlled Transmission
`of Traffic Flow Information", filed Sep. 10, 2002, Sandeep Jain.
`* cited by examiner
`
`Cloudflare - Exhibit 1023, page 2
`
`Cloudflare - Exhibit 1023, page 2
`
`

`

`U.S. Patent
`
`Dec. 25, 2007
`
`Sheet 1 of 9
`
`US 7,313,100 B1
`
`NETWORK
`6
`
`2
`
`14
`
`ACCOUNTING
`SERVER
`10
`
`NETWORK
`MONITOR
`4
`4
`
`fr18
`
`FIG.
`
`16
`
`REAL-TIME
`PACKET
`ANALYZER
`12
`
`Cloudflare - Exhibit 1023, page 3
`
`Cloudflare - Exhibit 1023, page 3
`
`

`

`waled 'SA
`
`et7
`
`N ye
`N
`CD
`0
`--.1
`
`6 Jo Z Jams
`
`to ooi`£i£`L sri
`
`4
`
`PORT
`26C
`
`PORT
`26D
`
`16
`
`14
`
`---
`
`--- — NETWORK ----
`
`... ...1...
`
`...--
`
`/
`/
`ROUTER
`20A
`
`24 --\
`si1/4 _r- 25A
`0
`
`6
`
`\
`
`\
`
`1
`
`/
`
`ROUTER
`20B
`
`Ila.—
`
`/
`
`...---
`
`____, --
`
`-----
`
`•-.
`
`25
`
`21B
`
`21A
`
`NETWORK MONITOR
`
`PORT
`26A
`
`PORT
`26B
`
`CONTROL
`UNIT
`28
`
`c271fr1/4.
`
`FILTER
`30
`
`a3 ACCOUNTING
`MODULES
`32
`
`FIG. 2
`
`Cloudflare - Exhibit 1023, page 4
`
`Cloudflare - Exhibit 1023, page 4
`
`P
`

`

`U.S. Patent
`
`Dec. 25, 2007
`
`Sheet 3 of 9
`
`US 7,313,100 B1
`
`4
`
`7
`NETWORK MONITOR
`
`33
`
`21A—\
`
`21BC
`
`14
`
`16 =
`
`IFC
`34
`•
`•
`
`IFC
`34
`ACCOUNTING
`SC
`36
`•
`•
`•
`ACCOUNTING
`SC
`36
`ENCRYPTION
`SC
`38
`
`TUNNNEL SC
`40
`
`CONTROL UNIT
`42
`
`.or
`
`FIG. 3
`
`Cloudflare - Exhibit 1023, page 5
`
`Cloudflare - Exhibit 1023, page 5
`
`

`

`U.S. Patent
`
`Dec. 25, 2007
`
`Sheet 4 of 9
`
`US 7,313,100 B1
`
`PACKET PACKET
`STREAM STREAM
`A
`
`ACCOUNTING
`SC
`36
`
`TUNNEL
`SC
`40
`
`CONTROL
`UNIT
`42
`
`52 0.
`
`58
`
`\6 2
`
`ENCRYPTION
`SC
`38
`
`SAMPLED
`FLOW
`RECORDS TRAFFIC
`
`FIG. 4
`
`Cloudflare - Exhibit 1023, page 6
`
`Cloudflare - Exhibit 1023, page 6
`
`

`

`U.S. Patent
`
`Dec. 25, 2007
`
`Sheet 5 of 9
`
`US 7,313,100 B1
`
`36
`
`ACCOUNTING SERVICE
`CARD
`
`INTERFACE
`70
`
`ACCOUNTING
`UNIT
`72
`
`FIG. 5
`
`Cloudflare - Exhibit 1023, page 7
`
`Cloudflare - Exhibit 1023, page 7
`
`

`

`Plalud *S11
`
`6 J° 9 JaotiS
`
`OOI`£I£`L Sfl
`
`100
`
`102
`A
`100
`
`102
`
`IFC
`82
`•
`•
`•
`IFC
`82
`ACCOUNTING
`SC
`84
`•
`•
`•
`ACCOUNTING
`SC
`84
`ENCRYPTION
`SC
`86
`
`TUNNNELSC
`88
`
`ROUTER
`
`80
`
`90
`
`4
`
`CONTROL UNIT
`
`FORWARDING
`ENGINE
`92
`
`ROUTING ENGINE
`94
`
`FORWARDING
`INFORMATION
`96
`
`4
`
`0
`
`ROUTING
`INFORMATION
`98
`
`FILTER
`99
`
`FIG. 6
`
`Cloudflare - Exhibit 1023, page 8
`
`Cloudflare - Exhibit 1023, page 8
`
`

`

`U.S. Patent
`
`Dec. 25, 2007
`
`Sheet 7 of 9
`
`US 7,313,100 B1
`
`ACCOUNTING SERVICE
`CARD
`
`7 84
`
`112
`
`INTERFACE
`
`115
`
`114
`
`118
`
`ACCOUNTING
`UNIT
`111
`
`BUFFER
`116
`
`TIMER
`117
`
`FIG. 7
`
`Cloudflare - Exhibit 1023, page 9
`
`Cloudflare - Exhibit 1023, page 9
`
`

`

`U.S. Patent
`
`Dec. 25, 2007
`
`Sheet 8 of 9
`
`US 7,313,100 B1
`
`y - 120
`
`RECEIVE NETWORK PACKETSAND
`GENERATE FIRST AND SECOND
`DUPLICATE PACKET STREAMS
`
`y- 122
`DISTRIBUTE PACKETS OF FIRST
`STREAM TO ACCOUNTING
`CARDS FOR CALCULATION OF
`FLOW RECORDS
`
`-134
`
`FILTER SECOND PACKET
`STREAM TO PRODUCE
`SAMPLED PACKET FLOWS
`
`y- 124
`
`136
`
`RECEIVE FLOW RECORDS AND
`ORIGINAL PACKETS FROM
`ACCOUNTING CARDS
`
`,-- 126
`v
`FORWARD PACKETS
`ACCORDING TO FORWARDING
`INFORMATION
`
`ANALYZE PACKET FLOWS
`
`138
`
`NO
`
`NETWORK
`CONDITION?
`
`1
`
`7-128
`
`YES
`
`140
`
`ANALYZE FLOW RECORDS
`
`UPDATE FORWARDING
`INFORMATION
`
`
`
`142
`
`FORWARD NETWORK ATTACK
`INFORMATION TO
`NEIGHBORING ROUTERS
`
`130
`
`NO
`
`SUSPICIOUS
`FLOWS?
`
`YES
`
`y- 132
`
`UPDATE FILTER TO INCLUDE
`SUSPICIOUS FLOWS
`
`FIG. 8
`
`Cloudflare - Exhibit 1023, page 10
`
`Cloudflare - Exhibit 1023, page 10
`
`

`

`U.S. Patent
`
`Dec. 25, 2007
`
`Sheet 9 of 9
`
`US 7,313,100 B1
`
`156
`
`158
`
`160
`
`150
`
`152
`
`0 .
`
`
`.
`c
`
`/
`/
`
`0 "
`
`
`
`151
`
`0 0
`
`0 0
`
`0
`
`0
`
`0 E
`
`
`i
`D
`
`11
`
`0 TEl
`El
`
`FIG. 9
`
`Cloudflare - Exhibit 1023, page 11
`
`0
`
`g D
`
`
`
`—
`
`154
`
`:
`11 I
`
`d
`
`IC 11
`
`CO
`
`1
`
`7-
`
`01 Iral II=
`
`Cloudflare - Exhibit 1023, page 11
`
`

`

`US 7,313,100 B1
`
`1
`NETWORK DEVICE HAVING ACCOUNTING
`SERVICE CARD
`
`TECHNICAL FIELD
`
`The invention relates to computer networks and, more
`particularly, to techniques for analyzing traffic flow within
`computer networks.
`
`BACKGROUND
`
`A computer network is a collection of interconnected
`computing devices that can exchange data and share
`resources. In a packet-based network, such as the Internet,
`the computing devices communicate data by dividing the
`data into small blocks called packets, which are individually
`routed across the network from a source device to a desti-
`nation device. The destination device extracts the data from
`the packets and assembles the data into its original form.
`Dividing the data into packets enables the source device to
`resend only those individual packets that may be lost during
`transmission.
`The packets are communicated according to a communi-
`cation protocol that defines the format of the packet. A
`typical packet, for example, includes a header carrying
`source and destination information, as well as a payload that
`carries the actual data. The de facto standard for communi-
`cation in conventional packet-based networks, including the
`Internet, is the Internet Protocol (IP).
`A system administrator or other user often makes use of
`a network analyzer to monitor network traffic and debug
`network problems. In general, a network analyzer is a tool
`that captures data from a network and presents the data to the
`user. The network analyzer typically allows the user to
`browse the captured data, and view summary and detail
`information for each packet. Accordingly, the user can view
`the network traffic flowing between devices on the network.
`The information collected during traffic flow analysis may
`be used for network planning, traffic engineering, network
`monitoring, usage-based billing and the like. Many conven-
`tional network analyzers, such as NetFlow, NeTraMet and
`FlowScan, use software applications to collect traffic flow
`information.
`The analyzers typically monitor and collect packets hav-
`ing routing information that matches criteria specified by the
`system administrator. The system administrator may specify,
`for example, source and destination Internet Protocol (IP)
`addresses, source and destination port numbers, protocol
`type, type of service (ToS) and input interface information.
`The analyzers typically collect packets matching the speci-
`fied criteria, and construct flow analysis diagrams. Conven-
`tional network analyzers often make use of sampling tech-
`niques to selectively sample the packets, and present a
`statistically generated view of the traffic within the network.
`Consequently, the statistics generated by the network ana-
`lyzer may not only be limited to specified flows, but may be
`relatively inaccurate.
`
`2
`In one embodiment, an apparatus comprises a set of
`interface cards to receive packets from a network, and a set
`of accounting service cards to calculate flow statistics for the
`packets. The apparatus further comprises a control unit to
`5 receive the packets from the interface cards and distribute
`the packets to the set of accounting service cards.
`In one embodiment, an accounting service card comprises
`an interface for insertion within a slot of a network device,
`to and an accounting unit to receive packets from the network
`device via the interface. The accounting unit calculates flow
`statistics based on the network packets.
`In another embodiment, a method comprises receiving
`packets from a network via an interface card of a network
`15 device, and distributing the packets to a set of accounting
`service cards of the network device. The method further
`comprises calculating with the accounting service cards flow
`statistics for the packets.
`In another embodiment, a method for computing flow
`20 statistics within an accounting service card of a network
`device comprises receiving packets from a control unit of a
`network router via an interface, and calculating flow statis-
`tics for the packets. The method further comprises output-
`ting a packet stream carrying the flow statistics and the
`25 received packets to the control unit for routing in accordance
`with muting information for the network.
`The techniques may provide one or more advantages. For
`example, according to the principles of the invention, mul-
`tiple accounting service cards may be added to easily scale
`30 the network monitor to support monitoring and accounting
`for higher bandwidth communication links. Depending upon
`processing power, two accounting service cards may be used
`to provide accounting for a single OC-3 communication
`link, while four cards and sixteen cards may be used to
`35 monitor OC-12 and 0C-48 links, respectively. As another
`example, eight accounting service cards may be used to
`monitor four OC-3 links. Additional accounting service
`cards may be used for purposes of redundancy to support
`continuous, uninterrupted packet processing and accounting
`40 in the event of card failure.
`Consequently, the flow analysis and packet intercept
`features may be readily integrated within a router for a
`packet-based network. The router may, for example, operate
`as a core router within the Internet to route packets received
`45 from high data rate communication links, such as OC-3,
`OC-12, OC-48, and greater communication links. The router
`may integrate accounting functionality to generate flow
`records for routed packets, as well as intercept features to
`capture packets for select packet flows. In this manner, the
`router can adjust routing functions based on the generated
`flow records and intercepted packets, thereby dynamically
`reacting to network events, such as Denial of Service (DOS)
`attacks and other network security violations.
`The details of one or more embodiments of the invention
`are set forth in the accompanying drawings and the descrip-
`tion below. Other features, objects, and advantages of the
`invention will be apparent from the description and draw-
`ings, and from the claims.
`
`50
`
`55
`
`SUMMARY
`
`60
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`In general, the invention is directed to techniques for
`monitoring and analyzing traffic flows within a network. A
`network monitor, in accordance with the principles of the
`invention, integrates accounting functionality for generation
`of flow statistics with packet intercept fimtionality to pro-
`vide a comprehensive traffic analysis environment.
`
`FIG. 1 illustrates an exemplary system in which a network
`monitor integrates accounting functionality for generation of
`65 flow records along with packet intercept functionality to
`provide a comprehensive traffic analysis environment in
`accordance with the principles of the invention.
`Cloudflare - Exhibit 1023, page 12
`
`Cloudflare - Exhibit 1023, page 12
`
`

`

`US 7,313,100 B1
`
`3
`FIG. 2 is a block diagram illustrating an example embodi-
`ment of a network monitor consistent with the principles of
`the invention.
`FIG. 3 is a block diagram illustrating another exemplary
`embodiment of a network monitor in further detail.
`FIG. 4 is a block diagram illustrating the flow of packets
`through the various components of a network monitor in
`accordance with the principles of the invention.
`FIG. 5 is a block diagram illustrating an example embodi-
`ment of an accounting service card in accordance with the
`principles of the invention.
`FIG. 6 is a block diagram illustrating an example embodi-
`ment of a router that incorporates accounting and intercept
`functionality.
`FIG. 7 is a block diagram illustrating another embodiment
`of an accounting service card.
`FIG. 8 is a flowchart illustrating operation of router that
`integrates traffic analysis and intercept features with routing
`functionality to dynamically react to network events, such as
`Denial of Service (DOS) attacks and other network security
`violations.
`FIG. 9 is a schematic diagram illustrating an exemplary
`embodiment of a network router that integrates traffic analy-
`sis and intercept features with routing functionality.
`
`DETAILED DESCRIPTION
`
`FIG. 1 illustrates an exemplary system 2 in which a
`network monitor 4 integrates accounting functionality for
`generation of flow records with packet intercept function-
`ality to provide a comprehensive traffic analysis environ-
`ment in accordance with the principles of the invention.
`Network monitor 4 is coupled to network 6 for monitoring
`network traffic. Network 6 may be formed by an intercon-
`nected group of autonomous systems, each representing an
`independent administrative domain having a variety of net-
`worked resources capable of packet-based communication.
`For example, network 6 may include servers, workstations,
`network printers and fax machines, gateways, routers, and
`the like. Each autonomous system within network 6 typi-
`cally includes at least one router for sharing routing infor-
`mation with, and forwarding packets to, the other autono-
`mous systems via communication links.
`The term "packet" is used herein to generally describe a
`unit of data communicated between resources in conform-
`ance with a communication protocol. The principles of the
`invention may be readily applied to a variety of protocols,
`such as the Transmission Control Protocol (TCP), the User
`Datagram Protocol (UDP), the Internet Protocol (IP), Asyn-
`chronous Transfer Mode, Frame Relay, and the like. Accord-
`ingly, "packet" is used to encompass any such unit of data,
`and may be interchanged with the term "cell," or other
`similar terms used in such protocols to describe a unit of data
`communicated between resources within the network.
`As described, network monitor 4 includes one or more
`accounting modules that generate accurate flow statistics for
`traffic within network 6. More specifically, network monitor
`4 captures packets from one or more links within network 6,
`and can generate flow statistics for each packet flow within
`the link. As network monitor 4 receives packets, the
`accounting modules associate the network packets with
`respective packet flows, and update the statistics for the
`packets flows. For example, the accounting modules may
`maintain an accurate packet count, byte count, source IP
`address, destination IP address, next hop IP address, input
`interface information, output interface information, total
`octets sent, flow start time, flow end time, source and
`
`4
`destination port numbers, TCP flags, IP type of service,
`originating AS, source address prefix mask bits, destination
`address prefix mask bits, and the like, for each packet flow.
`The accounting modules provide real-time accounting
`5 capabilities for maintaining accurate flow statistics for all of
`the packets received by network monitor 4. In particular, as
`described herein, the accounting modules can monitor and
`generate statistics for high traffic rates, even core traffic rates
`of the Internet, including OC-3, OC-12, OC-48, and higher
`to rates.
`Network monitor 4 outputs a stream of flow records 14
`that carry flow statistics for the captured packets. Network
`monitor 4 may, for example, output flow records 14 carrying
`accounting data for each flow, such as a number of packets,
`is a number of bytes, a time of capturing a first packet for the
`flow, a time of capturing a most recent packet for the flow,
`an incoming interface, an outgoing interface, a source/
`destination network mask, a source/destination Autonomous
`System (AS) number, and the like. Accounting server 10
`20 receives flow records 14, and updates an accounting system
`based on the flow records for further detailed analysis.
`In addition, network monitor 4 provides intercept capa-
`bilities that allow a real-time packet analyzer 12 to monitor
`specific packet flows within network 4. Network monitor 4
`25 outputs a stream of packets 16 to real-time packet analyzer
`12 for further analysis. The stream of packets 16 comprises
`a subset of the packets captured from network 6. In particu-
`lar, network monitor 4
`intercepts packets for one or more
`selected packet flows within network 4, and outputs the
`30 intercepted packets as a stream of packets 16. Packet ana-
`lyzer 12 receives the stream of packets 16, and analyzes the
`packets to identify any suspicious packet flows. For
`example, packet analyzer 12 may identify packet flows
`arising from Denial of Service (DOS) attacks and other
`35 network security violations.
`A system administrator may provide intercept information
`to network monitor 4
`that specifies a set of packet flows for
`which to capture packets. The system administrator may
`provide the intercept information directly, e.g., via a key-
`40 board, mouse or other input mechanism, to control inter-
`ception of packet flows. In addition, an administrator may
`remotely provide the routing information to network moni-
`tor 4 via a remote management protocol. In this manner, the
`administrator may selectively define the packet flows, and
`45 packets within a given flow, that are intercepted for analysis.
`Network monitor 4 may also control the stream of inter-
`cepted packets 16 based on feedback from accounting server
`10. More specifically, accounting server 10 may perform
`preliminary traffic analysis based on the flow records 14
`so received from network monitor 4, and provides filter infor-
`mation 18 to the network monitor to control the interception
`and forwarding of packets flows to packet analyzer 12 for
`further analysis. In this manner, network monitor 4 inte-
`grates accounting functionality for generation of flow
`55 records 14 along with packet intercept functionality to
`provide a comprehensive traffic analysis environment.
`Although illustrated as a stand-alone apparatus, the fea-
`tures of network monitor 4 may be integrated within a
`network device. For example, as described in detail below,
`60 the feature may be integrated within a router. Other network
`devices in which the features may be integrated include
`gateways, switches, servers, workstations, and the like.
`FIG. 2 is a block diagram illustrating in further detail an
`example embodiment of network monitor 4 coupled to
`65 communication links 24 of network 6. As illustrated, net-
`work 6 includes routers 20A, 20B ("routers 20") coupled via
`communication links 24. Routers 20 may comprise conven-
`Cloudflare - Exhibit 1023, page 13
`
`Cloudflare - Exhibit 1023, page 13
`
`

`

`US 7,313,100 B1
`
`5
`tional routers that forward packets in accordance with a
`topology of network 6. Communication links 24 may com-
`prise uni-directional optical links for carrying packets
`between routers 20 at high data rates, such as OC-3, 0C12,
`OC-48 and greater rates. Optical splitters 25A, 25B ("optical
`splitters 25") may be inserted within communication links
`24 to passively collect optical data transmitted and received
`between routers 20.
`Network monitor 4 includes two ports 26A, 26B for
`receiving the optical data 21A, 21B, respectively, and for-
`warding the data in digital form to control unit 28. As
`discussed in detail, control unit 28 merges the inbound data
`21A, 21B received from ports 26A, 26B, and digitally
`generates two identical packets streams 27A, 27B from the
`data. Control unit 28 applies filter 30 to packet stream 27A
`to selectively capture packet flows 16 for forwarding to
`packet analyzer 12 via output port 26C. In addition, control
`unit 28 distributes packets of the second stream 27B to
`accounting modules 32. Accounting modules 32 generate
`flow records 14 based on all of the packets of data stream
`27B, i.e., all of the packets received form optical splitters 25,
`and forward flow records 14 to accounting server 10 via
`output port 26D.
`Accounting modules 32 may buffer flow records 14 for a
`given packet flow until the flow "expires," i.e., when the
`accounting modules 32 detect inactivity for the flow for a
`configurable period of time, e.g., 30 minutes. Accounting
`modules 32 may periodically output batches of flow records
`14 for all flows that have recently expired, e.g., every fifteen,
`thirty or sixty seconds. For packet flows that remain active
`for long durations, accounting modules 32 may be config-
`ured to automatically expire the packet flows idler a defined
`duration, e.g., 30 or 60 minutes. Upon marking the active
`packet flow as expired, accounting modules 32 may output
`one or more flow records 14 for the packet flow, and may
`reset the statistics for the packet flow. Alternatively, account-
`ing modules may output flow records 114 without resetting
`the statistics for the active packet flow.
`FIG. 3 is a block diagram illustrating another exemplary
`embodiment of a network monitor 4. In the illustrated
`embodiment, network monitor 4 includes a chassis 33 for
`housing control unit 42. Chassis 33 has a number of slots
`(not shown) for receiving a set of cards, including interface
`cards (IFCs) 34, accounting service cards (ACCOUNTING
`SCs) 36, an encryption service card (ENCRYPTION SC) 38,
`and a tunnel service card (TUNNEL SC) 40. Each card may
`be inserted into a corresponding slot of chassis 33 for
`electrically coupling the card to control unit 42 via a bus,
`backplane, or other electrical communication mechanism.
`Interface cards 34 include ports for receiving inbound
`data 21 from optical splitters 25, and for outputting flow
`records 14 and intercepted packet flows 16. Accordingly,
`interface cards 34 include a number of ports (not shown) for
`coupling with communication links.
`Accounting service cards 36 each include one or more
`accounting modules that generate flow records based on
`packets received from control unit 42. Each accounting
`service card 36 may, for example, include one or more
`microprocessors, FPGAs, ASICs, or other components. As
`described, control unit 42 distributes packets to accounting
`service cards 36 for accounting and generation of flow
`records 14. In one embodiment, control unit 42 distributes
`the packets of a common flow to a common accounting
`service card 36. In other words, control unit 42 distributes
`packet flows across accounting service cards 36, and ensures
`that packets of any particular flow are distributed to a
`common one of accounting service cards 36. In this manner,
`
`6
`each of accounting service cards can generate complete flow
`records for the packet flows for which the card receives
`packets.
`In one embodiment, control unit 42 applies a hashing
`5 function to at least a portion of the header for each packet to
`ensure that packet flows are distributed across accounting
`service cards 36, and that packets of a packet flow are
`distributed to a common one of the accounting service cards
`36. Control unit 42 may apply a hashing function to at least
`to one of a source network address, a destination network
`address, and a communication protocol for the packet.
`Control unit 42 may apply the hash function to header
`information with each packet to generate a hash value, and
`distribute each packet to one of the accounting service cards
`15 36 based on the calculated hash values. Furthermore, por-
`tions of the header information may be selected to cause
`packet fragments associated with a common one of the
`network packet to be distributed to a common one of the
`accounting service cards. For example, layer 4 port infor-
`20 motion may be ignored, which may not be present for packet
`fragments.
`Multiple accounting service cards 36 may be added to
`easily scale network monitor 4 to support monitoring and
`accounting for higher bandwidth communication links. For
`25 example, depending upon processing power, two accounting
`service cards 36 may be used to provide accounting for a
`single OC-3 communication link, while four cards and
`sixteen cards may be used to monitor OC-12 and 0C-48
`links, respectively. As another example, eight accounting
`30 service cards 36 may be used to monitor four OC-3 links.
`Additional accounting service cards 36 may be used for
`purposes of redundancy to support continuous, uninter-
`rupted packet processing and accounting in the event of card
`failure.
`35 As described with respect to accounting modules 32 (FIG.
`2), accounting service cards 36 may output the flow records
`14 for a given packet flow when the flow "expires," i.e.,
`when the accounting service cards 36 detect inactivity for
`the flows for a configurable period. For example, accounting
`40 service cards 36 may make use of inactivity timers to
`determine when to output flow records. For packet flows that
`remain active for long durations, accounting service cards 36
`may be configured to automatically expire the packet flows
`after a defined duration, e.g., 30 or 60 minutes.
`If accounting server 10 and packet analyzer 12 are co-
`located with network monitor 4, control unit 42 may direct
`the flow records and intercepted packets directly to an
`appropriate output port of interface cards 34. In environ-
`ments where accounting server 10 and packet analyzer 12
`50 are located at remote destinations from network monitor 4,
`control unit 42 may make use of encryption service card 38
`and tunnel service card 40 to preserve security.
`Encryption service card 38 provides cryptographic func-
`tionality to network monitor 4. In particular, control unit 42
`55 may forward flow records generated by accounting service
`cards 36 to encryption service card 38 prior to forwarding to
`accounting server 10. In addition, control unit 42 may
`forward the intercepted packets for the select packet flows to
`encryption service card 38 for encryption prior to forward-
`60 ing to packet analyzer 12.
`Network monitor 4 may also include a network tunneling
`mechanism for relaying the flow records and intercepted
`packets through tunnels. Encryption service card 38 may
`provide IPSec tunnel, while tunnel service card 40 may
`65 provide GRE and IPIP tunnels. Tunnel service card 40
`aggregates traffic received from interface cards 34, and
`returns the traffic back to control unit 42 for output via
`Cloudflare - Exhibit 1023, page 14
`
`45
`
`Cloudflare - Exhibit 1023, page 14
`
`

`

`US 7,313,100 B1
`
`7
`interface cards 34. Control unit 42 may apply filter-based
`forwarding (FBF) to direct the returned traffic to the appro-
`priate output port of IFCs 34.
`FIG. 4 is a block diagram illustrating the flow of packets
`through the various components of a network monitor 50 in
`accordance with the principles of the invention. In the
`illustrated example, network monitor 50 monitors multiple
`communication links. In particular, network monitor 50
`collects transmit and receive packets for a first communi-
`cation link (labeled packet stream A in FIG. 1), and a second
`communication link (packet stream B). The first communi-
`cation link may, for example, comprise an OC-12 link, and
`the second communication link may comprise an OC-48
`link.
`Initially, control unit 42 receives packets streams A, B via
`separate monitoring ports (not shown). As described above,
`optical splitters may be used to passively collect packet
`streams A, B from the respective communication links.
`Control unit 42 distributes packet stream A, 13 to accounting
`service cards 36 for generation of a stream of packets 50
`carrying flow records. More specifically, accounting service
`cards 36 collect information from the packet flows within
`packet streams A, B and, based on the information, output
`packets 50 carrying flow records to control unit 42.
`If encryption is enabled, control unit 42 forwards packet
`stream 50 as packet stream 52 to carry the flow records to
`encryption card 38 for encryption. Encryption card 38
`encrypts each incoming packet 52, and returns a stream of
`encrypted packets 54 to control unit 42. Control unit 42
`forwards the encrypted packets carrying flow records 14 to
`accounting server 10 via an output port of one or more of
`interface cards 34.
`Simultaneous with the above-described accounting pro-
`cess, control unit 42 mirrors and filters each of the incoming
`packets of incoming packet streams A, B to produce packet
`streams A', B'. Control unit 42 may, for example, buffer
`incoming packets for packet streams A, B, and digitally copy
`each buffered packet to internally mirror packets streams A,
`B. Control u