`
`
`
`
`
`
`
`United States Patent
`(12)
`US 7,385,924 Bl
`(10) Patent No.:
`
`
`
`
`
`
`
`(45) Date of Patent:
`Jun. 10, 2008
`Riddle
`
`
`
`
`US007385924B1
`
`
`
`
`
`
`
`
`(54) ENHANCED FLOW DATA RECORDS
`
`
`
`
`INCLUDING TRAFFIC TYPE DATA
`
`(75)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Inventor: Guy Riddle, Los Gatos, CA (US)
`:
`.
`:
`
`
`
`
`
`
`(73) Assignee: Packeteer, Inc., Cupertino, CA (US)
`
`
`
`
`
`
`
`
`(*) Notice:
`Subject to any disclaimer, the term ofthis
`
`
`
`
`patent isacbyby952 adjusted under 35
`S.C.
`y
`ays.
`
`
`
`
`
`
`(21) Appl. No.: 10/676,383
`
`
`
`
`Sep. 30, 2003
`
`(22)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`3/2004 Calabrez etal.
`6,701,359 Bl
`
`
`
`
`
`5/2004 Yamadaet al. «0.0.0.0... 370/238
`6,738,352 BL*
`
`
`
`
`
`9/2004 Kimuraet al.
`6,798,763 Bl
`
`
`
`
`
`
`
`... 370/229
`5/2005 Phaal
`tenes
`6,894,972 BL*
`
`
`
`
`
`
`
`
`we. 726/13
`7,120,931 B1* 10/2006 Cheriton ........
`
`
`
`
`
`
`
`7,193,968 B1*
`3/2007 Kapoor et al... 370/235
`
`
`
`
`
`2002/0122427 Al
`9/2002 Kamenisky et al.
`
`
`
`bonebdeses ‘i 00s peotal
`
`
`
`
`2003/0112764 Al
`6/2003. Gaspard etal.
`
`
`* cited by examiner
`:
`
`
`
`Primary Examiner—Doris H. To
`
`
`
`Assistant Examiner—lan N. Moore
`
`
`
`
`(74) Attorney, Agent, or Firm—Mark J. Spolyar
`
`
`
`
`
`(57)
`ABSTRACT
`
`
`
`
`
`Method:
`d
`flow-based
`di
`d
`
`
`
`
`
`
`to a How-based,
`systems
`ethod’s, apparatuses anc’
`directed’
`
`
`
`
`
`
`traffic-classification-aware data collection and reporting sys-
`
`
`
`
`
`
`
`tem that combine flow-based data collection technologies
`
`
`
`
`
`
`
`with enhancedtraffic classification functionality to allow for
`
`
`
`
`
`
`
`
`analysis and reporting into aspects of network operations
`
`
`
`
`
`
`
`
`that prior art systems cannot provide. Embodiments provide
`
`
`
`
`
`
`
`
`enhanced views into the operation of computer network
`
`
`
`
`
`infrastructures to facilitate monitoring, administration, com-
`
`
`
`
`
`
`
`
`pliance and other tasks associated with networks. When a
`
`
`
`
`
`
`
`traffic flow terminates, a traffic monitoring device emits a
`
`
`
`
`
`
`
`flow data record (FDR) containing measurements variables
`and other attributes for an individual flow. A data collector
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`gathers the flow data records and enters them into a data-
`
`
`
`
`
`
`
`
`base. A network management application can then query the
`
`
`
`
`
`
`
`database with selected commandsto derive reports charac-
`
`
`
`
`
`
`terizing operation of the network suitable to diagnose prob-
`lems or view conditions associated with the network.
`
`
`
`
`
`
`
`
`
`
`
`26 Claims, 7 Drawing Sheets
`
`
`
`Filed:
`
`
`Int. Cl.
`
`
`
`(2006.01)
`GOIR 31/08
`
`
`
`
`
`
`
`(52) U.S. Ch cee 370/235; 370/252; 370/389
`
`
`
`
`
`
`(58) Field of Classification Search ............. 370/395.3,
`
`
`
`
`
`
`370/395.31. 395.5. 230. 428. 429. 419. 235
`
`
`
`
`
`
`
`370/359 381 389 392 952255 305 34,
`
`
`
`370/395 .52: 74 11203209 21: 709238244.
`
`
`
`
`
`709/223 24: 796/12 13: 713/152
`,
`
`
`
`
`
`
`
`See application file for complete search history.
`
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`
`
`(51)
`
`
`
`
`
`(56)
`
`
`
`
`
`
`
`4/1990 Sriram
`4,914,650 A
`
`
`
`
`
`5,828,846 A * 10/1998 Kirby etal. we. 709/238
`
`
`
`
`6,003,077 A
`12/1999 Bawdenetal.
`
`
`
`6,023,456 A
`2/2000 Chapman et al.
`
`
`
`6,046,980 A
`4/2000 Packer
`
`
`
`6,219,050 Bl
`4/2001 Schaffer
`
`
`
`
`
`6,285,660 Bl
`9/2001 Ronen
`
`
`
`
`
`6,397,359 Bl
`5/2002 Chandraet al.
`
`
`
`
`
`6,584,467 Bl
`6/2003 Haughtetal.
`
`
`
`
`6,681,232 Bl
`1/2004 Sistanizadehetal.
`
`
`
`
`
`
`
`
`
`
`
`50
`
`
`
` Traffic Monitoring
`
`——Traffic Monitoring Module
`
`Identifier
`iiFDREmitter:
`
`
`
`
`
`
`
`
`
`
`
`Device
`
`
`
`
`
`
`
`Packet
`Processor
`
`
`
`Splunk Inc.—Exhibit 1020 Page 1
`
`Splunk Inc. Exhibit 1020 Page 1
`
`

`

`
`U.S. Patent
`
`
`
`
`
`Jun. 10, 2008
`
`
`
`
`
`Sheet 1 of 7
`
`
`
`US 7,385,924 B1
`
`
`
`
`
`50 Traffic Monitoring
`
`
`Device
`
`
`
`
`
`40
`
`
`
`
`Fig.1A
`
`
`
`Splunk Inc.
`
`Exhibit1020
`
`Page 2
`
`Splunk Inc. Exhibit 1020 Page 2
`
`

`

`
`U.S. Patent
`
`
`
`
`
`Jun. 10, 2008
`
`
`
`
`Sheet 2 of 7
`
`
`
`US 7,385,924 B1
`
`
`
`
`
`50
` Computer
`
`
`
`Network
`
`Splunk Inc.
`
`Exhibit1020
`
`Page 3
`
`Splunk Inc. Exhibit 1020 Page 3
`
`

`

`
`U.S. Patent
`
`
`
`
`
`Jun. 10, 2008
`
`
`
`
`Sheet 3 of 7
`
`
`
`US 7,385,924 B1
`
`
`
`Bandwidth
`
`Management
`
`Device
`
`
`
`a
`
`-.
`
`130
`
`
`
`40
`
`J
`
`44
`
`
`
` Data
`
`
`Collector
`
`
`Traffic
`saoMeasurement
`
`
`
`
`
`
`137
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Flow
`Database
`
`
`
`
`
`
`
`
`
`
`
`
`Data Packet
`
`In
`
`
`
`
`Classification
`Engine
`
`
`Database
`138
`
` Management
`
`
`
`Information Base
`
`
`139
`
`
`
`
`
`
`
`140
`
`
`
`
`
`
`
`
`
`
`Out
`
`
`
`
`
`Packet
`Flow Control
`
`
`Processor
`Module
`
`
`
`
`
`
`131
`132 Fig.3
`
`
`Host
`Database
`
`
`
`
`
`
`
`
`FDR Emitter
`
`
`
`
`Data Packet
`
`
`
`
`Splunk Inc.
`
`Exhibit1020
`
`Page 4
`
`Splunk Inc. Exhibit 1020 Page 4
`
`

`

`
`U.S. Patent
`
`
`
`
`
`Jun. 10, 2008
`
`
`
`
`Sheet 4 of 7
`
`
`
`US 7,385,924 B1
`
`
`
`
`Receive Data
`
`Packet
`
`
`
`Flow
`
`
`
`
`
`
`
`Variables
`
`
`wah.FlowObject
`
`Construct
`
`
`
`
`
`
`Fetch/Update
`
`
`Flow Object
`
`Changes
`To Flow?
`
`
`
`Record Flow
`
`Measurement
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Splunk Inc.
`
`Exhibit1020
`
`Page5
`
`Splunk Inc. Exhibit 1020 Page 5
`
`

`

`
`U.S. Patent
`
`
`
`
`
`Jun. 10, 2008
`
`
`
`
`Sheet 5 of 7
`
`
`
`US 7,385,924 B1
`
`
`FDR Emitter
`
`Process
`
`
`
`
`
`Copy Flow
`
`
`Attributes and
`
`Measurement
`
`Variables
`
`
`
`
`Compose &
`
`
`Store Flow Data
`
`Record
`
`
`
`
`Increment
`
`FDR Counter
`
`
`
`256
`
`
`
`250
`
`
`
`252
`
`
`
`254
`
`
`
`
`
`
`
`
`
`@ Threshold?
`
`
`
`
`
`
`Yes
`
`
`
`258
`
`
`
`
`
`Get Global
`
`MIB Variables
`
`
`
`
`
`
`Compose
`FDR Message
`
`
`
`
`Fig.5
`
`
`
`
`
` FDR Counter
` 262
`
`
`
`
`
`
`
`
`Transmit FDR
`
`
`
`Record to Data
`
`Collector
`
`
`
`264
`
`
`
`
`Reset FDR
`
`Counter
`
`
`
`Splunk Inc.
`
`Exhibit1020
`
`Page6
`
`Splunk Inc. Exhibit 1020 Page 6
`
`

`

`
`U.S. Patent
`
`
`
`Jun. 10, 2008
`
`
`
`
`
`Sheet 6 of 7
`
`
`
`
`US 7,385,924 B1
`
`
`
`
`Receive Data
`
`Packet
`
`
`
`
`Block?ae—
`
`Construct
`
`
`
`Emitter
`
`Control
`
`
`
`
`
`
`
`
`Fetch/Update
`
`
`Control Block
`
`Changes
`To Flow?
`
`
`
`Write Traffic
`
`
`Class & Policies
`
`
`into Control Block
`
`
`
`
`
`Pass Packet to
`
`
`Flow Control
`
`Module (P)
`
`
`
`Record Flow
`
`Measurement
`
`Variables
`
`
`Identify
`
`Traffic Class
`
`
`
`
`Notify FDR
`
`
`
`
`Splunk Inc.
`
`Exhibit1020
`
`Page7
`
`Splunk Inc. Exhibit 1020 Page 7
`
`

`

`
`U.S. Patent
`
`
`
`
`
`Jun. 10, 2008
`
`
`
`
`Sheet 7 of 7
`
`
`
`US 7,385,924 B1
`
`302
`
`
`
`
`Receive
`
`Message
`
`
`
`
`
`
`
`
`Mapping
`
`Message?
`
`
`
`308
`
`
`
`
`
`
`
`Message in
`
`Mapping Table
`
`
`
`
`
`306 Store Mapping
`
`Message
`
`
`
`
`
`
`
`
`
`FDR
`
`Message?
`
`
`
`
`
`Discard
`
`
`
`
`
`
`
`
`
`
` 310
`
`
`Store Message
`
`Header in Header
`
`Table
`
`
`
`
`
`312
`
`
`
`Store FDRsin
`
`
`FDR Tables
`
`
`
`Splunk Inc.
`
`Exhibit1020
`
`Page8
`
`Splunk Inc. Exhibit 1020 Page 8
`
`

`

`
`1
`ENHANCED FLOW DATA RECORDS
`
`
`
`
`INCLUDING TRAFFIC TYPE DATA
`
`
`
`
`
`
`
`US 7,385,924 B1
`
`COPYRIGHT NOTICE
`
`
`
`
`
`
`
`
`
`
`
`
`
`A portion of the disclosure of this patent document
`
`
`
`
`
`
`contains material which is subject to copyright protection.
`
`
`
`
`
`
`
`
`
`The copyright owner has no objection to the facsimile
`
`
`
`
`
`
`
`reproduction by anyone of the patent documentor the patent
`
`
`
`
`
`
`
`disclosure as it appears in the Patent and Trademark Office
`
`
`
`
`
`
`
`
`
`patent file or records, but otherwise reserves all copyright
`
`
`rights whatsoever.
`
`CROSS-REFERENCE TO RELATED
`
`APPLICATIONS AND PATENTS
`
`
`
`
`
`
`20
`
`
`
`
`
`
`
`
`This application makes reference to the following com-
`
`
`
`
`
`
`
`
`monly owned U.S. patent applications and patents, which
`
`
`
`
`
`
`
`
`are incorporated herein by reference in their entirety for alt
`
`purposes:
`
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 08/762,828 now U.S. Pat.
`
`
`
`
`
`
`
`No. 5,802,106 in the name of Robert L. Packer, entitled
`
`
`
`
`
`
`
`
`“Method for Rapid Data Rate Detection in a Packet Com-
`
`
`
`
`
`
`25
`munication Environment Without Data Rate Supervision;”
`
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 08/970,693 now U.S. Pat.
`
`
`
`
`
`
`
`No. 6,018,516, in the name of Robert L. Packer, entitled
`
`
`
`
`
`“Method for Minimizing Unneeded Retransmission of Pack-
`
`
`
`
`
`
`ets in a Packet Communication Environment Supporting a
`
`
`
`
`Plurality of Data Link Rates;”
`
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 08/742,994 now U.S. Pat.
`
`
`
`
`
`
`
`No. 6,038,216, in the name of Robert L. Packer, entitled
`
`
`
`
`
`
`
`
`“Method for Explicit Data Rate Control in a Packet Com-
`
`
`
`
`
`
`munication Environment without Data Rate Supervision;”
`
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 09/977,642 now U.S. Pat.
`
`
`
`
`
`
`
`No. 6,046,980, in the name of Robert L. Packer, entitled
`
`
`
`
`
`
`
`“System for Managing Flow Bandwidth Utilization at Net-
`
`
`
`
`
`
`
`
`work, Transport and Application Layers in Store and For-
`
`
`ward Network;”
`
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 09/106,924 now U.S. Pat.
`
`
`
`
`
`
`
`
`No. 6,115,357, in the name of Robert L. Packer and Brett D.
`
`
`
`
`
`
`
`Galloway, entitled “Method for Pacing Data Flow in a
`
`
`Packet-based Network;”
`
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 09/046,776 now U.S. Pat.
`
`
`
`
`
`
`
`
`No. 6,205,120, in the name of Robert L. Packer and Guy
`
`
`
`
`
`
`Riddle, entitled “Method for Transparently Determining and
`
`
`
`
`
`
`Setting an Optimal Minimum Required TCP Window Size;”
`
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 09/479,356 now U.S. Pat.
`
`
`
`
`
`
`
`No. 6,285,658, in the name of Robert L. Packer, entitled
`
`
`
`
`
`
`
`“System for Managing Flow Bandwidth Utilization at Net-
`
`
`
`
`
`
`
`
`work, Transport and Application Layers in Store and For-
`
`
`ward Network;”
`
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 09/198,090 now U.S. Pat.
`
`
`
`
`
`
`
`
`No. 6,412,000, in the name of Guy Riddle and Robert L.
`
`
`
`
`
`
`Packer, entitled “Method for Automatically Classifying
`
`
`
`
`
`Traffic in a Packet Communications Network;”
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 09/198,051, now aban-
`
`
`
`
`
`
`
`
`doned, in the name of Guy Riddle, entitled “Method for
`
`
`
`
`
`
`Automatically Determining a Traffic Policy in a Packet
`
`
`Communications Network;”
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 09/206,772, now US.
`
`
`
`
`
`
`
`
`Pat. No. 6,456,630, in the name of Robert L. Packer, Brett
`
`
`
`
`
`
`
`
`
`
`D. Galloway and Ted Thi, entitled “Method for Data Rate
`
`
`
`
`
`Control for Heterogeneous or Peer Internetworking;”
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 10/039,992, now U.S.
`
`
`
`
`
`
`
`
`Pat. No. 7,032,072, in the name of Michael J. Quinn and
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`
`2
`
`
`
`
`
`
`
`Mary L. Laier, entitled “Method and Apparatus for Fast
`
`
`
`
`Lookup of Related Classification Entities in a Tree-Ordered
`
`
`Classification Hierarchy;”
`
`
`
`
`
`
`U.S. patent application Ser. No. 10/108,085, currently
`
`
`
`
`
`
`
`pending, in the name of Wei-Lung Lai, Jon Eric Okholm,
`
`
`
`
`
`
`and Michael J. Quinn, entitled “Output Scheduling Data
`
`
`
`
`
`Structure Facilitating Hierarchical Network Resource Allo-
`
`
`cation Scheme;”
`
`
`
`
`
`
`
`
`USS. patent application Ser. No. 10/155,936 now U.S. Pat.
`
`
`
`
`
`
`
`No. 6,591,299, in the name of Guy Riddle, Robert L. Packer,
`
`
`
`
`
`
`
`and Mark Hill, entitled “Method For Automatically Classi-
`
`
`
`
`
`
`fying Traffic With Enhanced Hierarchy In A Packet Com-
`
`
`munications Network;”
`
`
`
`
`
`
`U.S. patent application Ser. No. 10/236,149, currently
`
`
`
`
`
`
`
`pending, in the name of Brett Galloway and George Powers,
`
`
`
`
`
`entitled “Classification Data Structure enabling Multi-Di-
`mensional Network Traffic Classification and Control
`
`
`
`
`
`
`Schemes;”
`
`
`
`
`
`
`U.S. patent application Ser. No. 10/453,345, currently
`
`
`
`
`
`
`pending, in the name of Scott Hankins, Michael R. Morford,
`
`
`
`
`
`
`and Michael J. Quinn, entitled “Flow-Based Packet Cap-
`
`
`ture;” and
`
`
`
`
`
`
`U.S. patent application Ser. No. 10/611,573, currently
`
`
`
`
`
`
`pending, in the name of Roopesh Varier, David Jacobson,
`
`
`
`
`
`
`and Guy Riddle, entitled “Network Traffic Synchronization
`Mechanism.”
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIELD OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`
`The present invention relates to computer networks and,
`
`
`
`
`
`
`
`more particularly,
`to methods, apparatuses and systems
`directed to data collection schemesthat allow for enhanced
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`informational queries relating to the operation of computer
`network environments.
`
`
`
`BACKGROUND OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Efficient allocation of network resources, such as avail-
`
`
`
`
`
`
`able network bandwidth, has becomecritical as enterprises
`
`
`
`
`
`increase reliance on distributed computing environments
`
`
`
`
`
`
`and wide area computer networks to accomplish critical
`
`
`
`
`
`
`
`tasks. The widely-used TCP/IP protocol suite, which imple-
`ments the world-wide data communications network envi-
`
`
`
`
`
`
`
`
`
`
`
`
`
`ronment called the Internet and is employed in manylocal
`
`
`
`
`
`
`
`area networks, omits explicit supervisory function over the
`
`
`
`
`
`
`
`
`rate of data transport over the various devices that comprise
`
`
`
`
`
`
`
`the network. While there are certain perceived advantages,
`
`
`
`
`
`
`this characteristic has the consequence of juxtaposing very
`
`
`
`
`
`
`high-speed packets and very low-speed packets in potential
`
`
`
`
`
`
`conflict and produces certain inefficiencies. Certain loading
`
`
`
`
`conditions degrade performance of networked applications
`and can even cause instabilities which could lead to over-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`loads that could stop data transfer temporarily. The above-
`
`
`
`
`
`
`identified U.S. patents and patent applications provide
`
`
`
`
`
`explanations of certain technical aspects of a packet based
`
`
`
`
`telecommunications network environment, such as Internet/
`
`
`
`
`
`
`
`Intranet technology based largely on the TCP/IP protocol
`
`
`
`
`
`
`suite, and describe the deployment of bandwidth manage-
`
`
`
`
`
`
`ment solutions to monitor and/or manage network environ-
`
`
`
`
`
`
`ments using such protocols and technologies.
`
`
`
`
`
`
`
`The managementof such networks requires regular moni-
`
`
`
`
`
`
`
`toring and collection of data characterizing various attributes
`
`
`
`
`
`
`
`
`
`of the network,
`its operation and/or the traffic flowing
`
`
`
`
`
`
`
`
`
`through it. For example, Cisco Systems, Inc. of San Jose,
`
`
`
`
`
`
`
`
`Calif. offers a feature set of data monitoring and collection
`
`
`
`
`
`
`technologies in connection withits routers, called Netflow®.
`
`
`
`
`
`
`
`
`
`
`The Cisco JOS® NetFlow feature set allowsfor the tracking
`Splunk Inc.
`Exhibit 1020
`Page 9
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Splunk Inc. Exhibit 1020 Page 9
`
`

`

`
`
`
`
`
`
`
`
`
`US 7,385,924 B1
`
`
`
`
`
`20
`
`
`4
`
`
`
`
`
`
`Center™ leverages the powerful network utilization and
`
`
`
`
`
`application performance statistics available in Packet-
`
`
`
`
`
`
`shaper® bandwidth management devices andoffers a cen-
`
`
`
`
`
`
`tralized reporting platform to monitor and manage large
`
`
`
`
`
`deployments efficiently by streamlining collection, colla-
`
`
`
`
`
`
`
`tion, storage, analysis, and distribution of measuredstatis-
`tics.
`
`
`
`
`
`
`
`While the measurement engineis sufficient to achieveits
`
`
`
`
`
`
`
`
`intended purpose, some useful data for analyzing network
`
`
`
`
`
`
`
`usage and/or diagnosing problemsis not available histori-
`
`
`
`
`
`
`
`
`cally, but is only kept in memory while the PacketSeeker,
`
`
`
`
`
`PacketShaper or other bandwidth management device is
`
`
`
`
`
`
`
`
`
`In particular,
`the reports on “top talkers” and
`running.
`
`
`
`
`
`
`
`
`
`“traffic history” are not available for specific intervals in the
`
`
`
`
`
`
`
`
`
`past nor available after the device crashes, possibly due to
`
`
`
`
`
`
`
`
`
`some kind of attack or power outage. Furthermore, data
`
`
`
`
`
`
`
`maintained by the measurement engine,
`is generally not
`
`
`
`
`
`
`
`
`flow-based, and cannot answer questionslike “which clients
`
`
`
`
`
`
`
`are running port scanners.” Furthermore, as discussed
`
`
`
`
`
`
`
`above, NetFlow records characterize individual flows; how-
`
`
`
`
`
`
`
`
`ever, standard NetFlow records cannot answer such ques-
`
`
`
`
`
`
`
`
`tions or others requiring classification of flows beyond the
`
`
`
`
`attributes maintained by NetFlow records.
`
`
`
`
`
`
`
`
`
`
`In light of the foregoing, a need in the art exists for
`
`
`
`
`
`
`
`methods, apparatuses and systems that enable a flow-based,
`
`
`
`
`
`
`traffic-classification-aware data collection and reporting sys-
`
`
`
`
`
`
`
`
`tem. A need further exists in the art for methods, apparatuses
`
`
`
`
`
`
`
`and systems allowing for enhanced informational queries
`
`
`
`
`
`relating to the operation of networks. Embodiments of the
`
`
`
`
`
`
`present invention substantially fulfill these needs.
`SUMMARY OF THE INVENTION
`
`
`
`
`
`3
`
`
`
`
`
`
`
`
`of individual IP flows as they are received at a router or
`
`
`
`
`
`
`
`switching device. According to the technology, after a flow
`
`
`
`
`
`
`
`has terminated, a suitably configured router or switch gen-
`
`
`
`
`
`
`erates a NetFlow record characterizing various attributes of
`
`
`
`
`
`
`
`the flow. The NetFlow record is ultimately transmitted as a
`
`
`
`
`
`
`
`datagram to a NetFlow Data Collector that stores and,
`
`
`
`
`
`
`
`optionally, filters the record. A NetFlow Record includes a
`
`
`
`
`
`
`
`
`
`variety of attributes, such as source and destination IP
`
`
`
`
`
`
`
`
`
`addresses, packet count, byte count, start and end time
`
`
`
`
`
`
`
`stamps, source and destination TCP/UDPports, Quality of
`
`
`
`
`
`
`Service attributes, and routing-related information (e.g.,
`
`
`
`
`
`
`
`
`nexthop and Autonomous System (AS) data). Such Net-
`
`
`
`
`
`
`
`
`
`Flow® records are similar to call records, which are gener-
`
`
`
`
`
`
`
`
`ated after the termination of telephone calls and used by the
`
`
`
`
`
`
`
`
`
`telephone industry as the basis of billing for long distance
`
`
`
`calls, for example.
`
`
`
`
`
`
`
`Most network devices maintain data characterizing utili-
`
`
`
`
`
`
`
`zation, operation and/or performance of the network
`
`
`
`
`
`
`
`
`devices, and/or the network on which the devices operate, in
`
`
`
`
`
`
`
`
`limited, volatile memory, rather than using persistent storage
`
`
`
`
`
`
`
`
`(e.g., hard disks or other non-volatile memory). Conse-
`
`
`
`
`
`
`quently, network management applications commonly use
`
`
`
`
`
`
`
`the Simple Network Management Protocol (SNMP)to poll
`
`
`
`
`
`
`
`network devices (using the Management Information Base
`
`
`
`
`
`
`
`
`(MIB)associated with the network device) at regular time
`
`
`
`
`
`
`
`
`intervals and maintain the sampled raw data in a persistent
`
`
`
`
`
`
`
`data store. The network managementapplication, such as a
`
`
`
`
`
`
`
`
`
`reporting package, then processes the raw data to allow for
`
`
`
`
`
`
`
`
`
`the creation of reports derived from the raw data detailing
`
`
`
`
`
`
`
`
`operation and/or performance of the device and/or the
`
`
`
`
`
`
`network. ManagementInformation Basestypically contain
`
`
`
`
`
`
`
`low-level information characterizing the operation of the
`
`
`
`
`
`
`
`network device, such as the number of bytes or packets
`
`
`
`
`
`
`encountered on an interface, and do not provide information
`
`
`
`
`
`concerning the characteristics of data flows.
`
`
`
`
`
`
`Using a reporting package, a network administrator may
`
`
`
`
`
`
`
`
`
`then analyze the data to yield information about the perfor-
`manceor utilization of the network and/or network devices
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`associated with the network. Indeed, Various applications
`
`
`
`
`
`
`
`
`
`
`can then access the Data Collector to analyze the data for a
`
`
`
`
`
`
`variety of purposes, including accounting, billing, network
`
`
`
`
`
`
`
`planning, traffic engineering, and user or application moni-
`
`
`
`
`
`
`toring. There are public-domain implementations of collec-
`
`
`
`
`
`
`
`
`tors for standard NetFlow records. These are, however,
`
`
`
`
`
`
`
`
`unable to answer questions such as “which hosts are running
`
`
`
`
`
`
`
`
`
`the busiest Kazaa (or other peer-to-peer file sharing) serv-
`
`
`
`
`
`
`
`
`
`
`ers” (as NetFlow records are not suitable for analyzing and
`
`
`
`
`
`
`
`
`
`classifying network traffic that does not use registered IP
`
`
`port numbers).
`
`
`
`
`
`
`
`Packeteer, Inc. of Cupertino, Calif. develops bandwidth
`
`
`
`
`
`
`
`monitoring, management, and reporting software and sys-
`
`
`
`
`
`
`tems. Its PacketSeeker® systems and PacketShaper® band-
`
`
`
`
`
`
`
`width management devices, among other things, provide
`
`
`
`
`
`
`“application aware” monitoring of network traffic enabling
`
`
`
`
`
`
`
`classification of network traflic flows on a per application
`
`
`
`
`
`
`basis. The Packetshaper® bandwidth management device
`
`
`
`
`
`
`includes functionality allowing for classification of network
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`traffic based on information from layers 2 to 7 of the OSI FIG. 1A is a functional block diagram showingatraffic
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`reference model. As discussed in the above-identified pat-
`monitoring device according to an embodiment of the
`
`
`
`
`
`
`
`
`
`ents and patent applications,
`the bandwidth management
`present invention.
`
`
`
`
`
`
`
`
`
`
`
`
`device includes a measurement engine operative to record or
`FIG. 1B is a functional block diagram illustrating a
`
`
`
`
`
`
`
`
`
`
`
`
`maintain numerictotals of a particular measurement variable
`computer network environment including a bandwidth man-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`at periodic intervals on a traffic classification basis. The
`agement device according to an embodimentof the present
`
`
`
`
`
`
`invention.
`
`bandwidth management device further includes a manage-
`
`
`
`
`
`
`
`
`
`
`
`
`
`ment information base including standard network objects
`FIG. 2 is an functional block diagram illustrating a
`
`
`
`
`
`
`
`
`
`
`
`
`
`maintaining counts relating, for example, to the operation of
`computer network environment including a bandwidth man-
`
`
`
`
`
`
`
`
`
`
`
`
`its network interfaces and processors. Packeteer’s Report-
`agement device and a data collector.
`SplunkInc.
`
`25
`
`
`
`30
`
`
`
`35
`
`
`
`40
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`
`
`
`
`
`
`
`The present invention provides methods, apparatuses and
`
`
`
`
`systemsdirected to a flow-based,traffic-classification-aware
`
`
`
`
`
`
`
`data collection and reporting system. Embodiments of the
`
`
`
`
`
`
`
`present invention combine flow-based data collection tech-
`
`
`
`
`
`
`nologies with enhancedtraffic classification functionality to
`
`
`
`
`
`
`
`
`
`allow for analysis and reporting into aspects of network
`
`
`
`
`
`
`
`
`operations that prior art systems cannot provide. Embodi-
`
`
`
`
`
`
`
`
`ments of the present invention provide deeper insight into
`
`
`
`
`
`
`
`
`the operation of computer networks and the application
`
`
`
`
`
`
`traffic traversing the networks. Embodiments of the present
`
`
`
`
`
`
`
`
`invention provide enhanced views into the operation of
`
`
`
`
`
`
`computer network infrastructures to facilitate monitoring,
`
`
`
`
`
`
`
`administration, compliance and other tasks associated with
`
`
`
`
`
`
`
`
`networks. In one embodiment, when a traffic flow termi-
`
`
`
`
`
`
`
`
`nates, a traffic monitoring device emits a flow data record
`
`
`
`
`
`(FDR) containing measurements variables, classification
`
`
`
`
`
`
`
`information, and other attributes for an individual flow. A
`
`
`
`
`
`
`
`
`
`
`data collector gathers the flow data records and enters them
`
`
`
`
`
`
`
`into a database. A network managementapplication can then
`
`
`
`
`
`
`
`
`query the database with selected commandsto derive reports
`
`
`
`
`
`characterizing operation of the network suitable to diagnose
`
`
`
`
`
`
`
`problems or view conditions associated with the network.
`DESCRIPTION OF THE DRAWINGS
`
`
`
`
`Exhibit 1020
`
`Page 10
`
`Splunk Inc. Exhibit 1020 Page 10
`
`

`

`
`
`US 7,385,924 B1
`
`
`
`
`
`
`
`
`DESCRIPTION OF PREFERRED
`
`
`EMBODIMENT(S)
`
`
`
`25
`
`30
`
`35
`
`40
`
`
`
`6
`5
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`memory 76, such as a hard disk drive or other suitable
`FIG. 3 is a functional block diagram setting forth the
`
`
`
`
`
`
`
`
`
`
`
`
`memory device, such writable CD, DVD,or tape drives. In
`functionality in a bandwidth management device according
`
`
`
`
`
`
`
`
`
`
`
`
`one embodiment, traffic monitoring device 30 collects and
`to an embodimentof the present invention.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`transmits flow data records to a remote, persistent datastore,
`FIG. 4 is a flow chart diagram providing a method,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`for example, in datagrams, XML messages and the like.
`according to an embodiment of the present
`invention,
`
`
`
`
`
`
`
`
`
`
`
`directed to the processing of packets. FIGS. 1B and2illustrate an operating environment where
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG.5 is a flow chart diagram showing a method, accord-
`traffic monitoring device 30 is a bandwidth management
`
`
`
`
`
`
`
`
`
`
`
`ing to an embodimentof the present invention, directed to
`device 130 (see discussion below).
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`composing and transmitting flow data records to a data
`As FIGS. 1A, 1B and 2 show,the traffic monitoring device
`collection node.
`
`
`
`
`
`
`
`
`
`30 (or bandwidth management device 130), in one embodi-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 6 is a flow chart diagram illustrating a method
`ment, is disposed on the link between a Local area network
`directed to enforcement of bandwidth utilization controls on
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`40 and router 22. In other embodiments, multiple traffic
`
`
`
`
`
`
`
`
`
`
`
`network traffic traversing an access links.
`monitoring devices can be disposedat strategic points in a
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 7 is a flow chart diagram providing a method
`given network infrastructure to achieve various objectives.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`directed to processing messages including flow data records.
`In addition, packet monitoring device 30 need notbe directly
`
`
`
`
`
`
`
`
`
`connected to the link between two network devices, but may
`
`
`
`
`
`
`
`
`
`also be connected to a mirror port. In addition, the traflic
`
`
`
`
`
`
`monitoring functionality described herein may be deployed
`
`
`
`
`
`
`
`
`in multiple network devices and used in redundant network
`
`
`
`
`
`
`topologies by integrating the networktraffic synchronization
`
`
`
`
`
`
`
`functionality described in U.S. application Ser. No. 10/611,
`
`
`573, above.
`
`
`
`
`A. Flow-Based Traffic Monitoring
`
`
`
`
`
`
`
`Asdiscussedherein,traffic monitoring device 30 is opera-
`
`
`
`
`
`
`
`
`
`tive to detect or recognize flows between end systems,
`
`
`
`
`
`
`
`
`
`classify the data flows based on one or moreflow attributes
`
`
`
`
`
`
`
`
`and, upon the termination of individual flows, compose flow
`
`
`
`
`
`
`
`
`data records including data fields characterizing one or more
`attributes associated with the individual flows. The flow data
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`records, in one embodiment, are ultimately transmitted to a
`
`
`
`
`
`
`
`
`data collector 44 whichstores the data in a database allowing
`
`
`
`
`
`
`
`applications to query the database to generate reports char-
`
`
`
`
`
`
`acterizing the operation of the network in a variety of ways
`
`
`
`
`
`
`
`
`
`that were not possible prior to the invention described
`
`
`
`
`
`
`herein. FIG. 4 illustrates a method, according to an embodi-
`
`
`
`
`
`
`
`ment of the present invention, directed to a flow-aware
`
`
`
`
`
`
`
`
`
`process that classifies flows and notifies a flow data record
`
`
`
`
`
`
`
`
`emitter that a flow has ended. FIG. 5 provides a method,
`
`
`
`
`
`
`
`
`according to an embodiment of the present
`invention,
`
`
`
`
`
`
`
`directed to composing flow data records and transmitting a
`
`
`
`
`
`
`
`plurality of flow data records in a datagram to a remote data
`collector 44.
`
`
`
`
`
`
`
`
`
`As FIG.4 illustrates, a packet processor 82 receives a data
`
`
`
`
`
`
`
`
`packet (102) and determines whether a flow object has
`
`
`
`
`
`
`
`
`
`
`already been created for the flow to which the data packetis
`
`
`
`
`
`
`
`
`a part (104). A flow object is a data structure includingfields
`
`
`
`
`
`
`
`
`whose values characterize various attributes of the flow,
`
`
`
`
`
`
`
`including source and destination IP addresses, port numbers,
`
`
`
`
`
`
`
`
`
`
`traffic type identifiers and the like. A flow object can also
`
`
`
`
`
`
`
`
`include other attributes, such as packet count, byte count,
`
`
`
`
`
`
`
`
`
`
`first packet time, last packet time, etc. If a flow object is not
`
`
`
`
`
`
`
`found, packet processor 82 constructs a new flow object
`
`
`
`
`
`
`
`(106). Packet processor 82 then determines whether the
`
`
`
`
`
`
`
`
`received packetis part of an existing flow or a new data flow
`
`
`
`
`
`
`
`
`(108). In one embodiment, flows are generally TCP and
`
`
`
`
`
`
`
`
`
`UDPflows. However, any suitable transport layer flow can
`
`
`
`
`
`
`
`
`
`be recognized and detected. In one embodiment, flows are
`
`
`
`
`
`
`
`
`identified based on the following flow attributes: 1) source
`
`
`
`
`
`
`
`IP address, 2) destination IP address, 3) source port number,
`
`
`
`
`
`
`
`
`
`4) destination port number, and 5) protocol (derived from the
`
`
`
`
`
`
`
`“protocol”field in IPv4 headers, and the “NextHeader”field
`
`
`
`
`
`
`
`
`
`
`in IPv6 headers). One skilled in the art will recognize that
`
`
`
`
`
`
`
`flows can be identified in relation to a variety of attributes
`
`
`
`
`
`
`
`
`and combinations of attributes. In addition, methods for
`
`
`
`
`
`
`
`
`determining new data flows and assigning packets to exist-
`
`
`
`
`
`
`
`
`
`
`
`ing data flows are well known in the art and also depend on
`SplunkInc.
`Exhibit 1020
`Page 11
`
`20
`FIG.1A illustrates a basic network environment in which
`
`
`
`
`
`
`
`
`
`
`
`
`an embodiment of the present invention operates. FIG. 1A
`
`
`
`
`
`
`
`
`showsa first network device 40, such as a hub, switch or
`
`
`
`
`
`
`
`router, interconnecting two end-systems (here, client com-
`
`
`
`
`
`
`
`
`
`
`puter 42 and host 44). FIG. 1A also provides a second
`
`
`
`
`
`
`
`network device 22, such as a router, operably connected to
`
`
`
`
`
`
`
`network cloud 50, such as an open, wide-area network. As
`
`
`
`
`
`
`
`
`FIG. 1A shows, packet traffic monitoring device 30 com-
`
`
`
`
`
`
`
`
`
`prises traffic monitoring module 75, and first and second
`
`
`
`
`
`
`
`
`network interfaces 71, 72, which operably connecttraffic
`
`
`
`
`
`
`monitoring device 30 to the communications path between
`first network device 40 and second network device 22.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Traffic monitoring module 75 generally refers to the func-
`
`
`
`
`
`
`
`tionality implemented by traffic monitoring device 30. In
`
`
`
`
`
`
`one embodiment, traffic monitoring module 75 is a combi-
`
`
`
`
`
`
`
`nation of hardware and software, such as a central process-
`
`
`
`
`
`
`
`
`
`ing unit, memory, a system bus, an operating system and one
`
`
`
`
`
`
`
`or more software modules implementing the functionality
`
`
`
`
`
`
`
`described herein. In one embodiment, trafic monitoring
`
`
`
`
`
`
`
`module 75 includes a packet processor 82, a traffic type
`
`
`
`
`
`
`
`
`
`
`identifier 84, and a flow data record emitter 86. In one
`
`
`
`
`
`
`embodiment, the packet processor 82 is operative to process
`
`
`
`
`
`
`
`data packets, such as storing packets in a buffer structure,
`
`
`
`
`
`
`
`
`
`
`detecting new data flows, and parsing the data packets for
`
`
`
`
`
`
`
`various attributes (such as source and destination addresses,
`
`
`
`
`
`
`
`
`and the like) and maintaining one or more measurement
`variables or statistics in connection with the flows. The
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`traffic type identifier 84, as discussed more fully below,is
`
`
`
`
`
`
`
`
`
`operative to classify data flows based on one or more
`attributes associated with the data flows. The flow data
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`record emitter 86 is operative to compose flow data records
`
`
`
`
`
`
`
`
`
`characterizing the data flows that traverse