00000001
`
` 1000000000
`
`USOU731310031
`
`(12}
`
`United States Patent
`Turner et at.
`
`(10) Patent No.:
`
`(45} Date of Patent:
`
`US 7,313,100 Bl
`Dec. 25, 2007
`
`(54)
`
`(75)
`
`NETWORK DEVICE HAVING ACCOUNTING
`SERVICE CARD
`
`6.182.146 Bl
`6.321.338 Bl’“
`
`1:"2001 Graham-Cumming.
`11-'200|
`Porras el al.
`
`Jr.
`726-"2.
`
`Inventors: Stephen W Turner. Menlo Park. CA
`(US): IIsien-Chttng Won. Fremont. CA
`(US); Sanjay Kalra. San Jose. CA
`(US); Truman Joe. Mountain View. CA
`(US); Wendy R Cartee. Los Altos. CA
`(US)
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`W0
`W0
`
`8-"1998
`WO 98366332 AI “‘
`W0 2084920 AB "‘ 10;:2002
`
`(73)
`
`Assigncc: Juniper Networks, Inc.. Sunnyvale.
`CA (US)
`
`OTHER PUBLICATIONS
`
`(*1
`
`Notice:
`
`Subject to any disclaimer. the term of this
`patent
`is extended or adjusted under 35
`U.S.C. 154(1)) by 1034 days.
`
`Weaver. AL“. et al.. "A Real-Time Monitor for Token Ring Net-
`works.” Military Conununications Conference. 1989. MILCOM
`‘89. Oct. 1989. vol. 3. pp. 794—798.*
`
`(21)
`
`Appl. No.: 107228.150
`
`(22)
`
`Filed:
`
`Aug. 26, 2002
`
`(51)
`
`Int. CI.
`11041. 72/26
`
`(52)
`
`U.S. Cl.
`
`(2006.01)
`
`3707253;3707244:3707252;
`3707392
`
`(58)
`
`(56)
`
`Field of Classification Search ................ 3707235.
`3707242—244. 250. 252. 253, 389. 392. 396.
`3707469. 471: 7091223. 224. 229
`See application lilc for complete search history.
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3.962.681 A
`4.032.899 A
`4.600.319 A
`5.408.539 A
`5.490.252 A ‘3
`5.509.123 A
`5.568.471 A “‘
`6.011.795 A
`6.018.765 A
`6.148.335 A ’°
`
`6-'1976 Requa et al.
`6:"1977 Jenny et a1.
`71986 Everett. Jr.
`431995 Finlay el al.
`2:"1996 Macera et a1.
`451996 Dobbins ct a1.
`10.-'1996 Hershey ct a].
`“2000 Varghese et a].
`1"2000 Durana et a1.
`11-'2000 Haggard et al.
`
`7093249
`
`370-245
`
`709924
`
`(Continued)
`
`(‘hi P113111
`Primary Examiner
`Assistant Examiner"- -Donald 1. Mills
`
`[74} glimmer. Agent. or Firm—Shumaker & Sieifert PA.
`
`(57)
`
`ABSTRACT
`
`A network device integrates accounting functionality for
`generation of flow statistics with packet intercept function-
`ality to provide a comprehensive traffic analysis environ-
`ment. The device comprises a set of network interface cards
`to receive packets from a network, and a set of accounting
`service cards to calculate [low statistics for the packets. The
`device further comprises a control unit to receive the net-
`work packets from the interface cards and distribute the
`packets to the set of accounting service cards. The account-
`ing service card comprises an interface for insertion within
`a slot of a network device. Accounting service cards may be
`added to easily scale the network device to support higher
`bandwidth conununication links. such as OC—3. OC—12.
`CCU-48 and higher rate links. Additional accounting service
`cards may be used for purposes of redtuldancy to suppon
`continuous. uninterrupted packet processing and accounting
`in the event oi'a card failure.
`
`24 Claims, 9 Drawing Sheets
`
`
`
`
`
`SAMPLE]!
`FLOW
`RECORDS TWFE
`
`Palo Alto Networks V. Sable Networks
`
`IPR2020-01712
`
`EX1023
`
`EX1023
`Palo Alto Networks v. Sable Networks
`IPR2020-01712
`
`

`

`US 7,313,100 B1
`
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`OTHER PUBLICATIONS
`
`6.392.996 Bl
`6.499.088 Bl
`6.563.796 131 ‘V
`6.590.898 Bl
`6.594.268 Bl
`6.598.034 Bl
`6.735.201 Bl
`6.751.663 Bl
`6.826.713 Bl
`6.983.294 B2 ’3
`6.985.956 B2 “
`7.114.008 B2
`200230141343 Al
`2003.?”0005145 Al
`20033000769 Al
`2003.30214913 Al
`
`5.02002
`1232002
`5.92003
`7.52003
`7.52003
`7.52003
`5’2004
`6.42004
`1 l 32004
`1.32006
`132006
`932005
`1032002
`1:200}
`552003
`1 1.52003
`
`Hjalmtysson
`chler et a].
`Saito
`Uzun
`Aukia et a].
`Kloth
`
`Mahajan et a1.
`Farrell et a].
`
`Beasley et 3].
`Jones et at.
`Luke or at.
`
`Jungck et al.
`Bays
`Bollard
`McCollom et al.
`Kan et a].
`
`.. 370-"252
`
`. 707.-"202
`. 70932 29
`
`.11.. “Performance Evaluation for Distributed System
`Dini. P. et
`Components.“ Proceedings of IEEE Second lnlemational Workshop
`on Systems Management. Jun. 1996. pp. 20-293“
`Integrated Services Adapter. 2000. Cisco Systems. Data Sheet. pp.
`[—6.
`1101:3151rW.cisoo.cowwarp-“publicfccw’pdfi faa’ivaaiiasvaa!
`prodlita'ism2_ds.pdl‘.
`“'l'hc CAIDA Web Site.“ m.caida.org:". 2000.
`“About Endace.“ wwsndacexom-‘l 2000.
`“Cisco IOS NetFlow.” wvtrw.cisco.comr'warpfpublicx'732fTechfmnpx'
`net flow-"index .sh tml. 2002.
`U .3. Appl. No. 105188.567. enlilled "Adaplive Network Flow
`Analysis". fiied Jul. 2. 2002. Scott Mackie.
`U.S. Appl. No. 10f228.l32. entitled “Adaptive Network Router".
`filed Aug. 26. 2002. Woo at a].
`U.S. Appl. No. 101928.114. entitled “Network Router Having Inte-
`grated Flow Accounting and Packet Interception". filed Aug. 26.
`2002. Woo el al.
`U.S. Appl. No. 107241.785. entitled “Rate-Controlled Transmission
`ofTraffic Flow Infonnation”. filed Sep. [0. 2002. Sandeep Jain.
`
`* cited by examiner
`
`

`

`US. Patent
`
`Dec. 25, 2007
`
`Sheet 1 OH)
`
`US 7,313,100 B1
`
`/ 2
`
`
`
`16
`
`
`REAL-TIME
`
`
`PACKET
`ACCOUNTING
`
`SERVER
`ANALYZER
`
`
`
`19
`E
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 25, 200’?
`
`Sheet 2 of 9
`
`US 7,313,100 B1
`
`02_._.ZDOOU<
`
`mun—3005—
`
`.65on
`
`:2:
`
`mm
`
`N.07.
`
`/
`
`/
`
`o\./..rf
`IIxmoafizt\\I11
`
`4mm/em
`
`
`
`
`

`

`US. Patent
`
`Dec. 25, 2007
`
`Sheet 3 OH)
`
`US 7,313,100 B1
`
`TUNNNELSC
`
`ACCOUNHNG
`SC
`
`ENCRYPNON
`SC
`
`FIG. 3
`
`CONTROLUNW
`
`E
`
`IO
`
`

`

`US. Patent
`
`Dec. 25, 2007
`
`Sheet 4 OH)
`
`US 7,313,100 B1
`
`PACKET PAC KET
`
`STREAM STREAM
`A
`B
`
`ACCOSLENTING
`a
`
`AI
`L.
`\T’
`+T55
`
`TUNNEL
`
`sc
`£
`
`1B
`
`52
`
`x—'
`55;
`
`ENCRYPTION
`so
`
`A &
`
`A ‘
`
`FLOW
`RECORDS
`
`SAMPLED
`TRAFFIC
`
`FIG.4
`
`

`

`US. Patent
`
`Dec. 25, 2007
`
`Sheet 5 OH)
`
`US 7,313,100 B1
`
`36
`
`
`IXCK3()LHH11hH313EHRVHCHE
`
`
`
`CARD
`
`
`
`UNW
`INTERFACE
`zg
`Z;
`
`
`ACCOUNNNG
`
`
`
`
`
`68
`
`FWC3.5
`
`

`

`US. Patent
`
`Dec. 25, 200’?
`
`Sheet 6 of 9
`
`US 7,313,100 B1
`
`
`
`20.5.4510".z.ZO_._.<_2¢O"_Z_
`
`
`
`ozzbomOz_oz<>>~_0u_
`
`mmmm
`
`Kuhn—E
`
`mm
`
`
`
`0.0."—
`
`3Um
`
`OZPZDOOO<
`
`Um
`
`.vm
`
`20Fm>mozm
`
`ow
`
`wm
`
`omezzzah
`
`fl
`
`
`
`mug—62mUZFZ=000<
`
`mzazu02:30”.Gzfiom<§z0u
`
`
`I..Hon:.3?
`
`so...
`
`NS.
`
`
`
`._._ZDAOKHZOO
`
`om
`\
`
`3E50”.3,
`
`
`
`
`
`
`

`

`US. Patent
`
`Dec. 25, 2007
`
`Sheet 7 OH)
`
`US 7,313,100 B1
`
`CARD
`
`
`
`ACCOUNTING SERVICE
`
`
`
`
`113
`
`
`\
`
`INTERFACE
`BUFFER
`
`
`
`112
`
`ACCOUNTING
`uurr
`
`11E.
`
`m
`
`
`
`114
`
`FIG. 7
`
`

`

`US. Patent
`
`Dec. 25, 2007
`
`Sheet 8 OH)
`
`US 7,313,100 B1
`
`RECEIVE NETWORK PACKETSAND
`GENERATE FIRST AND SECOND
`
`
`
`FILTER SECOND PACKET
`STREAM TO PRODUCE
`
`
`
`
`
`DISTRIBUTE PACKETS OF FIRST
`STREAM TO ACCOUNTING
`CARDS FOR CALCULATION OF
`FLOW RECORDS
`
`
`
`
`/ 124
`
`RECEIVE FLOW RECORDS AND
`ORIGINAL PACKETS FROM
`ACCOUNTING CARDS
`
`126
`
`DUPLICATE PACKET STREAMS
`SAMPLED PACKET FLOWS
`ANALYZE PAC KET FLOWS
`
`
`FORWARD PACKETS
`ACCORDING TO FORWARDING
`INFORMATION
`
`
`
`NETWORK
`
`
`CONDITION?
`128
`
`
`
`
`
`
`ANALYZE FLOW RECORDS
`
`UPDATE FORWARDING
`INFORMATION
`
`
`
`130
`
`US Pl CIOUS
`F LOWS?
`
`
`
`
`[—
`
`
`
`FORWARD NETWORK ATTACK
`INFORMATION TO
`NEIGHBORING ROUTERS
`
`
`
`
`
`
`UPDATE FILTER TO INCLUDE
`SUSPICIOUS FLOWS
`
`
`
`
`
`FIG. 8
`
`

`

`US. Patent
`
`Dec. 25, 200’?
`
`Sheet 9 of 9
`
`US 7,313,100 B1
`
`
`
`FIG. 9
`
`

`

`US ?,313,100 B1
`
`1
`NETWORK DEVICE HAVING ACCOUNTING
`SERVICE CARI)
`
`TECHNICAL FIELD
`
`The invention relates to computer networks and. more
`particularly, to techniques for analyzing traflic flow within
`computer networks.
`
`BACKGROU‘N D
`
`10
`
`A computer network is a collection of interconnected
`computing devices that can exchange data and share
`resources. In a packet-based network, such as the Internet.
`the computing devices communicate data by dividing the
`data into small blocks called packets. which are individually
`routed across the network from a source device to a desti-
`nation device. The destination device extracts the data from
`
`the packets and assembles the data into its original form.
`Dividing the data into packets enables the source device to
`resend only those individual packets that may be lost during
`transmission.
`
`The packets are communicated according to a communi-
`cation protocol
`that defines the format of the packet. A
`typical packet. for example.
`includes a header carrying
`source and destination information, as well as a payload that
`carries the actual data. The de facto standard for communi—
`cation in conventional packet-based networks, including the
`Internet, is the Intemet Protocol UP).
`A system administrator or other user often makes use of
`a network analyzer to monitor network traffic and debug
`network problems. In general. a network analyzer is a tool
`that captures data from a network and presents the data to the
`user. The network analyzer typically allows the user to
`browse the captured data, and view summary and detail
`information for each packet. Accordingly. the user can view
`the network trailic flowing between devices on the network.
`The information collected during traffic flow analysis may
`be used for network planning, traffic engineering, network
`monitoring, usage—based billing and the like. Many conven-
`tional network analyzers. such as NetF low, NeTraMet and
`lilowScan. use software applications to collect traffic flow
`information.
`
`The analyzers typically monitor and collect packets hav-
`ing routing information that matches criteria specified by the
`system administrator. The system administrator may specify,
`for example. source and destination Internet Protocol (IP)
`addresses. source and destination port numbers, protocol
`type. type of service (1'08) and input interface injomiation.
`The analyzers typically collect packets matching the speci-
`fied criteria. and construct flow analysis diagrams. Convert-
`tional network analyzers often make use of sampling tech-
`niques to selectively sample the packets. and present a
`statistically generated view of the traffic within the network.
`Consequently, the statistics generated by the network anav
`lyzer may not only be limited to specified flows, but may be
`relatively inaccurate.
`
`SWMARY
`
`In general. the invention is directed to techniques for
`monitoring and analyzing traffic flows within a network. A
`network monitor, in accordance with the principles of the
`invention, integrates accounting functionality for generation
`of flow statistics with packet intercept firntionality to pro-
`vide a comprehensive traffic analysis environment.
`
`3t]
`
`4t]
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`In one embodiment. an apparatus comprises a set of
`interface cards to receive packets front a network, and a set
`o'faccotuiting service cards to calculate flow statistics for the
`packets. The apparatus further comprises a control unit to
`receive the packets from the interface cards and distribute
`the packets to the set of accounting service cards.
`111 one embodiment, an accounting service card comprises
`an interface for insertion within a slot of a network device.
`
`and an accounting unit to receive packets from the hemork
`device via the interface. The accounting unit calculates flow
`statistics based on the network packets.
`In another embodiment. a method comprises receiving
`packets from a network via an interface card of a network
`device. and distributing the packets to a set of accounting
`service cards of the network device. The method further
`
`comprises calculating with the accounting service cards How
`statistics for the packets.
`In another embodiment, a method for computing flow
`statistics within an accounting service card of a network
`device comprises receiving packets from a control unit of a
`network router via an interface. and calculating fiow statis—
`tics for the packets. The method further comprises output-
`ting a packet stream carrying the flow statistics and the
`received packets to the control unit for routing in accordance
`with routing information for the network.
`The techniques may provide one or more advantages. For
`example. according to the principles of the invention. mul-
`tiple accounting service cards may be added to easily scale
`the network monitor to support monitoring and accounting
`for higher bandwidth communication links. Depending upon
`processing power, two accounting service cards may be used
`to provide accounting for a single OC-3 communication
`link. while four cards and sixteen cards may be used to
`monitor OCT-12 and 0048 links, respectively. As another
`example. eight accounting service cards may be used to
`monitor four 003 links. Additional accounting service
`cards may be used for purposes of redundancy to support
`continuous, uninterrttpted packet processing and accounting
`in the event of card failure.
`
`intercept
`the flow analysis and packet
`Consequently.
`features may be readily integrated within a router for a
`packet-based network. The router may, for example. operate
`as a core router within the Internet to route packets received
`from high data rate communication links, such as O'C-3.
`OCT-12. OC-48, and greater communication links. The router
`may integrate accounting functionality to generate flow
`records for routed packets. as well as intercept features to
`capture packets for select packet flows. In this manner, the
`router can adjust routing functions based on the generated
`flow records and intercepted packets, thereby dynamically
`reacting to network events. such as Denial of Service (DOS)
`attacks and other network security violations.
`The details of one or more embodiments of the invention
`
`are set forth in the accompanying drawings and the descrip«
`tion below. Other features. objects. and advantages of the
`invention will be apparent from the description and draw-
`ings. and from the claims.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`FIG. 1 illustrates an exemplary system in which a network
`monitor integrates accounting functionality for generation of
`flow records along with packet intercept functionality to
`provide a comprehensive traffic analysis environment
`in
`accordeuice with the principles of the invention.
`
`

`

`3
`
`4
`
`US ?,313,100 El
`
`FIG. 2 is a block diagram illustrating an example embodi-
`ment of a network monitor consistent with the principles of
`the invention.
`
`FIG. 3 is a block diagram illustrating another exemplary
`embodiment of a network monitor in further detail.
`FIG. 4 is a block diagram illustrating the [low of packets
`through the various components of a network monitor in
`accordance with the principles of the invention.
`FIG. 5 is a block diagram illustrating an example embodi-
`ment of an accounting service card in accordance with the
`principles of the invention.
`FIG. 6 is a block diagram illustrating an example embodi-
`tnent ofa router that incorporates accounting and intercept
`functionality.
`FIG. 7 is a block diagram illustrating another embodiment
`of an accounting service card.
`FIG. 8 is a flowchart illustrating operation of router that
`integrates traffic analysis and intercept features with routing
`functionality to dynamically react to network events. such as
`Denial of Service [DOS] attacks and other network security
`violations.
`
`FIG. 9 is a schematic diagram illustrating an exemplary
`embodiment of a network router that integrates trailic analy-
`sis and intercept features with routing functionality.
`
`DETAILED DESCRIPTION
`
`FIG. 1 illustrates an exemplary system 2 in which a
`network monitor 4 integrates accounting functionality for
`generation of [low records with packet intercept ftmction-
`ality to provide a comprehensive traffic analysis environ-
`ment in accordance with the principles of the invention.
`Network monitor 4 is coupled to network 6 for tnonitoring
`network traffic. Network 6 may be formed by an intercon-
`nected group of autonomous systems, each representing an
`independent administrative domain having a variety of net-
`worked resources capable of packet-based communication.
`For example, network 6 may include servers. workstations.
`network printers and fax machines. gateways. routers. and
`the like. Each autonomous system within network 6 typi-
`cally includes at least one router for sharing routing infor-
`mation with. and forwarding packets to, the other autono~
`mous systems via communication links.
`The term “packet“ is used herein to generally describe a
`unit of data communicated between resources in conform-
`
`ance with a communication protocol. The principles of the
`invention may be readily applied to a variety of protocols.
`such as the Transmission Control Protocol (TCP). the User
`Datagram Protocol (UDP), the Internet Protocol (IP), Asyn—
`chronous Transfer Mode, Frame Relay, and the like. Accord—
`ingly. “packet” is used to encompass any such unit ofdat'a.
`and may be interchanged with the temi “cell,“ or other
`similar terms used in such protocols to describe a unit ofdata
`communicated between resources within the network.
`As described. network monitor 4 includes one or more
`accounting modules that generate accttrate flow statistics for
`trafiic within network 6. More specifically, network monitor
`4 captures packets from one or more links within network 6,
`and can generate flow statistiCs for each packet flow within
`the link. As network monitor 4 receives packets,
`the
`accounting modules associate the network packets with
`respective packet flows. and update the statistics for the
`packets flows. For example, the accounting modules may
`maintain an accurate packet count, byte count, source IP
`address. destination IP address, next hop IP address. input
`interface information. output
`interface information.
`total
`octets sent,
`flow start
`time= flow end time. source and
`
`3t]
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`destination port numbers, TC P flags. IP type of service.
`originating AS. source address prefix mask bits. destination
`address prefix mask bits. and the like. for each packet flow.
`The accounting modules provide real-time accounting
`capabilities for maintaining accurate flow statistics for all of
`the packets received by network monitor 4. In particular. as
`described herein, the accounting modules can monitor and
`generate statistics for high traflic rates, even core traflic rates
`of the Internet, including OCT-3, (DC-12, 0(7-48, and higher
`rates.
`
`10
`
`Network monitor 4 outputs a stream of flow records 14
`that carry flow statistics for the captured packets. Network
`monitor 4 may, for example, output flow records 14 carrying
`accounting data for each flow. such as a number of packets,
`a number of bytes. a time of capturing a first packet for the
`flow. a time of capturing a most recent packet for the flow.
`an incoming interface. an outgoing interface. a source!
`destination network mask, a sourcefdestination Autonomous
`System (AS) number. and the like. Accounting server [0
`receives [low records 14. and updates an accounting system
`based on the flow records for further detailed analysis.
`In addition. network monitor 4 provides intercept capa-
`bilities that allow a real-time packet analyzer 12 to monitor
`specific packet flows within network 4. Network monitor 4
`outputs a stream of packets 16 to real-time packet analyzer
`12 for further analysis. The stream of packets 16 comprises
`a subset of the packets captured from network 6. ln particu-
`lar. network monitor 4 intercepts packets for one or more
`selected packet flows within network 4. and outputs the
`intercepted packets as a stream of packets l6. Packet ana-
`lyzer 12 receives the stream of packets l6. and analyzes the
`packets
`to identify any suspicious packet
`flows. For
`example. packet analyzer 12 may identify packet flows
`arising from Denial of Service (DOS) attacks and other
`network security violations.
`A system administrator may provide intercept infomiation
`to network monitor 4 that specifies a set of packet flows for
`which to capture packets. The system administrator may
`provide the intercept information directly. e.g.. via a key-
`board, mouse or other input mechanism. to control inter-
`ception of packet flows. In addition. an administrator may
`remotely provide the routing iuionnation to network moni—
`tor 4 via a remote management protocol. In this manner. the
`administrator may selectively define the packet flows. and
`packets within a given flow. that are intercepted for analysis.
`Network monitor 4 may also control the stream of inter-
`cepted packets 16 based on feedback from accounting server
`10. More specifically. accounting server 10 may perform
`preliminary traffic analysis based on the flow records 14
`received from network monitor 4. and provides filter infor—
`mation 18 to the network monitor to control the interception
`and forwarding of packets floats lo packet analyzer 12 for
`further analysis. In this manner. network monitor 4 inte-
`grates accounting functional ity for generation of flow
`records 14 along with packet
`intercept functionality to
`provide a comprehensive tralfic analysis environment.
`Although illustrated as a standalone apparatus. the fea—
`tures of network monitor 4 may be integrated within a
`network device. For example. as described in detail below.
`the feature may be integrated within a router. Other network
`devices in which the features may be integrated include
`gateways, switches. servers. workstations. and the like.
`FIG. 2 is a block diagram illustrating in further detail an
`example embodiment of network monitor 4 coupled to
`communication links 24 of network 6. As illustrated, net—
`work 6 includes routers 20A. 208 (“routers 20") coupled via
`communication links 24. Routers 20 may comprise conven-
`
`

`

`5
`
`6
`
`US ?,313,100 El
`
`tional routers that forward packets in accordance with a
`topology of network 6. Communication links 24 may com-
`prise uni-directional optical
`links
`for carrying packets
`between routers 20 at high data rates. such as OC-3. 0C12.
`0C -48 and greater rates. Optical splitters 25A. 25B (“optical
`splitters 25") may he insened within communication links
`24 to passively collect optical data transmitted and received
`between routers 20.
`Network monitor 4 includes two ports 26A, 268 for
`receiving the optical data 21A. 218. respectively. and for-
`warding the data in digital
`form to control unit 28. As
`discussed in detail. control unit 28 merges the inbound data
`21A. 2113 received from ports 26A, 261-3, and digitally
`generates two identical packets streams 27A. 27B from the
`data. Control unit 28 applies filter 30 to packet stream 27A
`to selectively capture packet flows 16 for forwarding to
`packet analyzer 12 via output port 26C. In addition. control
`unit 28 distributes packets of the second stream 278 to
`accounting modules 32. Accounting modules 32 generate
`llow records 14 based on all of the packets of data stream
`27B, i.e.. all ofthe packets received form optical splitters 25.
`and forward flow records 14 to accounting server 10 via
`output port 26!).
`Accounting modules 32 may bulfer flow records 14 for a
`given packet flow until the How “expires." i.e.. when the
`accounting modules 32 detect inactivity for the flow for a
`configurable period of time. e.g._. 30 minutes. Accounting
`modules 32 may periodically output batches of flow records
`14 for all flows that have recently expired, e.g.. every fifteen,
`thirty or sixty seconds. For packet flows that remain active
`for long durations, accounting modules 32 may be config-
`ured to automatically expire the packet flows after a defined
`duration. e.g.. 30 or 60 minutes. Upon marking the active
`packet flow as expired. accounting modules 32 may output
`one or more flow records 14 for the packet flow, and may
`reset the statistics for the packet flow. Alternatively. account—
`ing modules may output flow records 114 without resetting
`the statistics for the active packet flow.
`FIG. 3 is a block diagram illustrating another exemplary
`embodiment of a network monitor 4.
`[n the illustrated
`embodiment. network monitor 4 includes a chassis 33 for
`housing control unit 42. Chassis 33 has a number of slots
`(not shown) for receiving a set of cards, including interface
`cards [IFCs) 34. accounting service cards (ACCOUNTING
`SCs) 36. an encryption service card {ENCRYPTION SC) 38.
`and a tunnel service card ('IUNNEL SC) 40. Each card may
`be inserted into a corresponding slot of chassis 33 for
`electrically coupling the card to control unit 42 via a bus.
`backplane. or other electrical communication mechanism.
`Interface cards 34 include ports for receiving inbound
`data 21 from optical splitters 25. and for outputting flow
`records 14 and intercepted packet flows 16. Accordingly.
`interface cards 34 include a number of ports (not shown) for
`coupling with communication links.
`Accounting service cards 36 each include one or more
`accounting modules that generate flow records based on
`packets received from control unit 42. Each accounting
`service card 36 may. for example.
`include one or more
`microprocessors. l-‘PGAs, ASICs. or other components. As
`described. control unit 42 distributes packets to accounting
`service cards 36 for accounting and generation of flow
`records 14.
`in one embodiment. control unit 42 distributes
`the packets of a common flow to a common accounting
`service card 36. In other words. control unit 42 distributes
`packet flows across accounting service cards 36. and ensures
`that packets of any particular flow are distributed to a
`common one of accotmting service cards 36. In this manner.
`
`each ol'accounting service cards can generate complete flow
`records for the packet flows for which the card receives
`packets.
`In one embodiment. control unit 42 applies a hashing
`function to at least a portion of the header for each packet to
`ensure that packet flows are distributed across accounting
`service cards 36, and that packets of a packet flow are
`distributed to a common one ofthe accounting service cards
`36. Control ttnit 42 may apply a hashing function to at least
`one of a source network address. a destination network
`
`10
`
`for the packet.
`address. and a cormnunication protocol
`Control unit 42 may apply the hash function to header
`information with each packet to generate a hash value. and
`distribute each packet to one ofthe accounting service cards
`36 based on the calculated hash values. Furthermore. por-
`tions of the header inl'onnation may be selected to cause
`packet fragments associated with a common one of the
`network packet to be distributed to a common one of the
`accounting service cards. For example. layer 4 port infor-
`mation may be ignored. which may not be present for packet
`fragments.
`Multiple accotuIting service cards 36 may be added to
`easily scale network monitor 4 to support monitoring and
`accounting for higher bandwidth communication links. For
`example. depending upon processing power, two accounting
`service cards 36 may be used to provide accounting for a
`single OCT-3 communication link. while four cards and
`sixteen cards may be used to monitor OCT—12 and 0048
`links. respectively. As another example. eight accounting
`service cards 36 may be used to monitor four OCT-3 links.
`Additional accounting service cards 36 may be used for
`purposes of redundancy to support continuous. uninter-
`rupted packet processing and accounting in the event of card
`failure.
`As described with respect to accounting modules 32 (F 1G.
`2). accounting service cards 36 may output the flow records
`'14 for a given packet flow when the flow “expires." i.e..
`when the accounting service cards 36 detect inactivity for
`the flows for a configurable period. For example. accounting
`service cards 36 may make use of inactivity timers to
`determine when to output llow records. For packet [lows that
`remain active for long durations. accounting service cards 36
`may be configured to automatically expire the packet flows
`alter a defined duration. e.g.. 30 or 60 minutes.
`If accounting server It} and packet analyzer 12 are co-
`located with network monitor 4. control unit 42 may direct
`the flow records and intercepted packets directly to an
`appropriate output port of interface cards 34. In environ-
`ments where accounting server 10 and packet analyzer 12
`are located at remote destinations from network monitor 4.
`control unit 42 may make use of encryption service card 38
`and tunnel service card 40 to preserve security.
`[Encryption service card 38 provides cryptographic finic-
`tionalily to network monitor 4. In particular. control unit 42
`may forward flow records generated by accounting service
`cards 36 to encryption service card 38 prior to forwarding to
`accounting server 10.
`In addition, control unit 42 may
`forward the intercepted packets for the select packet flows to
`encryption service card 38 for encryption prior to forward-
`ing to packet analyzer 12.
`Network monitor 4 may also include a network tunneling
`mechanism for relaying the flow records and intercepted
`packets through tunnels. Encryption service card 38 may
`provide IPSec tunnel, while tunnel service card 40 may
`provide ORE and IPII’ tunnels. Tunnel service card 40
`aggregates traflic received from interface cards 34. and
`returns the traflic back to control unit 42 for output via
`
`3t]
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`7
`
`8
`
`US ?,313,100 B1
`
`interface cards 34. Control unit 42 may apply filter-based
`forwarding (FBI?) to direct the returned traffic to the appro-
`priate output port of lFCs 34.
`FIG. 4 is a block diagram illustrating the flow of packets
`through the various components of a network monitor 50 in
`accordance with the principles of the invention.
`In the
`illustrated example, network monitor 50 monitors multiple
`communication links.
`in particular, network monitor 50
`collects transmit and receive packets for a first communi-
`cation link (labeled packet stream A in FIG. 1). and a second
`commtutication link (packet stream B). The first communi-
`cation link may, for example. comprise an OC—12 link, and
`the second contmnlnication link may comprise an (DC—48
`link.
`
`Initially. control unit 42 receives packets streams A. B via
`separate monitoring ports (not shown). As described above.
`optical splitters may be used to passively collect packet
`streams A. B from the respective communication links.
`Control unit 42 distributes packet stream A. B to accounting
`service cards 36 for generation of a stream of packets 50
`carrying flow records. More specifically, accounting service
`cards 36 coilect information from the packet flows within
`packet streams A. B and. based on the information. output
`packets 50 carrying flow records to control unit 42.
`If encryption is enabled. control unit 42 forwards packet
`stream 50 as packet stream 52 to carry the flow records to
`encryption card 38 for encryption. Encryption card 38
`encrypts each incoming packet 52, and returns a stream of
`encrypted packets 54 to control unit 42. Control unit 42
`forwards the encrypted packets carrying flow records 14 to
`accounting server It] via an output port of one or more of
`interface cards 34.
`
`Simultaneous with the above-described accotmting pro-
`cess, control tutit 42 mirrors and filters each ofthe incoming
`packets of incoming packet streams A. B to produce packet
`streams A'. B'. Control unit 42 may. for example. buffer
`incoming packets for packet streams A. B, and digitally copy
`each bufl'ered packet to internally mirror packets streams A.
`B. Control unit 42 applies a filtering operation to the
`mirrored packet streams to produce packet streams A‘, 13'
`having intercepted packets for seiect packet flours. Conse-
`quently. packet streams A'. B' carry copies ofa subset of the
`packets within incoming packet streams A. B. respectively.
`Control unit 42 forwards packet streams A'. B‘ to tunnel
`service card 40 for aggregation and ioopback to control unit
`42 as aggregated packet stream 56. Finally. control unit 42
`applies filter—based forwarding (FBF) to forward aggregated
`packet stream 56 to packet analyzer 12 as output packet
`stream 16. More specifically. control unit 42 directs aggre-
`gated packet stream 56 to an appropriate output interface
`card as packet stream 16 for forwarding to packet anaiyzer
`12. If encryption is enabled. control unit may forward
`aggregated packet stream 56 to encryption service card 38 as
`packet stream 58, and may receive encrypted packet stream
`62 in return for forwarding to packet analyzer 12 as packet
`stream 16.
`
`FIG. 5 is a block diagram illustrating an example embodi-
`ment ofan accounting service card 36 in accordance with the
`principles of the invention. Accounting service card 36
`receives inbound packet stream 66 via interface 7!}. Interface
`70 may. for example, comprise a high~speed communication
`bus, back plane. switch fabric. or the like, to allow account—
`ing service card 36 to easily be inserted and removed from
`an interface siot within the chassis of network monitor 33.
`
`5
`
`1
`
`3t]
`
`4t]
`
`50
`
`55
`
`60
`
`65
`
`interface 70 allows multiple accounting
`In this fashion.
`service cards 36 to be added to network monitor chassis 33
`
`to support monitoring of high data rate communication
`links.
`
`Interface 70 forwards packet stream 66 to accounting unit
`72 for updating flow statistics. Interface 70 may forward
`each packet in its entirely, or may extract only those portions
`of the packets necessary for maintaining accurate flow
`statistics. Interface 70 may.
`for example. extract header
`information and forward the extracted header infomiation to
`
`accounting unit 72. In this manner. bandwidth efliciencics
`may be achieved mtween interface 70 and accounting unit
`72. The extracted header information may include inform