Live Traffic Analysis of TCP /IP Gateways t
`
`Phillip/'\ .. Porras
`porrasOcsl.sri.com
`Computer Science Ijal)oratory
`
`SRI I11Ler11ai.io11al
`333 n.avcnswood /\venue
`Menlo Park, CA 94025
`
`Alfonso Valdes
`avaldesOcsl.sri.com
`Electromagnetic ai1d Re111ote
`Sensing Laboratory
`SRI International
`;3;33 Rave1uiwood Avenue
`Menlo Park, C/\ 94025
`
`December 12 1997
`
`Abstract
`
`11ece.s.sary flo\VS demanded for user functionality, can be
`a nontrivial exercise [3].
`
`rve enumerate a variety of u:ays to extend buth sla(cid:173)
`t1st1cal and signature-based intrusiuu-de/eclion analy.~i.~
`techniques lo 1nonilo1· nft111ork traffiC:. Specifically, u1e
`pre.~f:nl techniques to analyze TCJ-'/lP packet strca111s
`that flow through network gateways for· 8i.l/nY of mali(cid:173)
`cious activity, no111naficiu·as failures, and othrr excep(cid:173)
`tional events. The intent is lo demon.~trnte, by exam.(cid:173)
`pie 1 the utility of introducing gatr111ay surveillance mech(cid:173)
`a11is1ns lo monitor net1vork traffic.
`ltJ.'e present this dis(cid:173)
`c1tssion of gateway surveillance 1uec!tanisrris a.~ com.ple(cid:173)
`nientary to the filteriug rnechanisms of a large enterprise
`nct·work, aud illuslrate the usefulness of surveillance in
`directly enhancing the security and stability of net111ork
`operations.
`
`1
`Introduction
`M cchanis1ns for p::i.rsing and filtering hostile exler(cid:173)
`
`nal nPtwork traffic [:.!, 4] that could reach internal
`net\\o·ork services have become widely accepted as prc(cid:173)
`requisitt:s for lirniting the exposure of internal net>vork
`W:lset.s V>'hile maintaining interco1111eclivity \Yith ext.er(cid:173)
`ual nel \\'or ks. The encoding of filteriug rules for p::i.cket(cid:173)
`or transport-layer con1municatiou should he enforced
`at entry pointR between internal network.sand external
`traffic. Developing filtering rules Lhat .strike an optin1al
`balance between the restrictiveness ner,esRary to sup(cid:173)
`pres.s the entry of unwanted Lraffic, while allowing the
`•t Th" W•>rk prl"sl"ntcdin this pa.per is cniT.,nLly funded by the
`Information Technoloi>;y Offict' of the Defense Advanced Rflsf!arch
`Projects Agf'rwy, under contract number F30!i02-96-C-U~!J4.
`
`ln additioll to intelligent. filtering, there have been
`various developmt:>nts in recent years in passive surveil(cid:173)
`lance mechanisms to 111onitor net,vork traffic for signs of
`malicious or anon1alous (e.g., potentially erroneous) ac(cid:173)
`tivity. Such tools atle1npl lo provi<le network adminis(cid:173)
`trators tin1ely in.sight into note\vorthy exceptional activ(cid:173)
`ity. Real-time monitoring pron1ises an added dimenRion
`of control and insight into the flow of traffic between
`the internal network and its external environment. The
`insight. gained through fielded network Lraflic monitors
`could also aid sites in enhancing the effectiveness of their
`fire\vall tillering rllles.
`
`Ilowever, traffic 111onitoring i.s not a free activity(cid:173)
`especially live Lraffic monitoring. In presenting our di.s(cid:173)
`cussion of nelwork analysis techniques, we fully real(cid:173)
`ize thf'. t'-Osts they in1ply -..vith respect t.o computational
`resources and hu1u1111 over.sight. For exan1ple, obtain(cid:173)
`ing the necessitry input for surveillanct: involves the de(cid:173)
`ployment of instrumentation to parse, filter, and for(cid:173)
`rnat event strean1s derived fro111 µotentially high-volun1e
`packet transmissions. Con1plex event analysis, response
`logic, and human n1anagc111ent of the analysis units also
`introduce costs. Clearly, the int.roduction of network
`Rllrveillance 1nechani::lrns on top of alrea.dy-deploycd pro(cid:173)
`tective traffic filters is an expense that requires justifica(cid:173)
`tion. In this paper, >ve outline the benefits of our tech(cid:173)
`nique~ and seek to persuade the reader that the costs
`can he worthwhile.
`
`In proceedings oft.he 1998 !SOC Symposium on Networ·k and Distributed Sy.•lf.m.B St:cu•·ity
`
`1 of 13
`
`Commerce Bancshares, Inc., et al.
`Exhibit 1004
`Page 1
`
`

`

`2 Toward Generalized Network
`Surveillance
`
`The Lechnique::; pre::iente<l in this paper arc extensions
`of earlier work hy SRI in developing analytical meth(cid:173)
`ods for detecting anomalous or known intrusive activ(cid:173)
`ity [1, 5, 12, 13]. Our earlier intrusion-detection ef(cid:173)
`forts in rh~veloping IDES (Intrusion DeterJ.ion F,xpert
`System) and later NIDES (!\ext-Generation Intrusion
`Detection Expert System) were oriented toward the
`Hurveillance of user-session an<l hosL-layer activity. This
`previous foe.us on session activity within host houn<l(cid:173)
`aries is understandable given that the primary input
`to intrusion-detection tools, audit data, is produced by
`1ncchauis1ns that tend to be locally ad1ninistcrcd 'll'ithin
`11 single host or domain. Ho¥lever, as the importanr.e
`of net,vork security has gro,vn, so too has the need to
`expand intrusion-detection technology to address net(cid:173)
`work infra;;;tructnre and service:s. Tn 011r r.urrent re(cid:173)
`search effort, E~IERALD (Event Monitoring Enabling
`llesponses to Anomalous Live Disturbances), we explore
`the extension of our intrusion-dctcctiuu uH..:tho<l~ tu the
`analysis of netv.·ork activity.
`
`Net>vork monitoring, in the context of fault detection
`and diagnosis for computer network and telecon1n1uni(cid:173)
`cation environrncnts, has becu ~tu<lied exie118ively by
`the network 1nanagement and alarm correlation commu(cid:173)
`nity [8, 11, 15, 16]. The high-volume distributed event
`correlation technology pron1oted in so1ne projects µru(cid:173)
`vides an exeellent foundation for building truly scalable
`net. \Vork-aware ,o;11rveillance technology for n1isuse. llow(cid:173)
`ever, these efforts focus prin1arily on the health au<l ~ta­
`t.us (fault detection and/or diagnosi:;,) or µerfor1nance of
`the target net.work, an<l <lo 11ot cover the detertion of in(cid:173)
`tentionally abu8ive traffic. Tn<leed, some simplifications
`in the fault analy,o;is a.nd diagnosis co111111unity (e.g., as(cid:173)
`sumptions of stateless correlation, which precludes even I.
`ordering; simplistic ti1ne-out 111etric:s for resetting the
`tracking of prol>le111s; ignoring individuals/sources re·
`sponsible for exceptional activity) do not translate well
`to a malicious environn1ent for detecting intrusions.
`
`Earlier work in the intrusion-detertion community
`atten1pting tu address the is,o;11e of network surveillance
`includes the Nr.:Lwork Security Monitor (NS.l\i), devel(cid:173)
`oped at UC Davi8 [6], and the Network Anon1aly De(cid:173)
`tection and Intrusion R,eporter (N AD Ill) [7], developed
`at Los Ala111us National Lahoratory (LANL). Both per(cid:173)
`fur111ed broadc:ast LA;"if packet 111onit.oring tu analyze
`traffic patterns for knov:n hostile or a1101nalou,o; activ(cid:173)
`ity. 1 Further research by UC Davis in the Distributed
`
`1Rcccnt product examples, 5uch ru; ASTM and Net Ranger,
`that follow the pa~~iv" pa<·ket 1non.itoring approach have 5ince
`
`[2il] and later
`Intrusion Detection Syste1n (DIDS)
`Graph-based Intrusion Detection Systern (GRIDS) [24]
`projects has attempted to extend intrusion monitor(cid:173)
`ing capabilities beyond LAN analysis, to provide rnulti(cid:173)
`LAN and very large-8cale netv.·ork coverage.
`
`'fhis paper takes a pragn1at.ic look at the issue
`of packet and/or datagra1n analysis ha.~ed on statis(cid:173)
`tical anomaly detection and signature-analysis tech-
`1uques.
`'!'his "'ork is being perforn1ed in the con(cid:173)
`text of SR.l's latest intru8ion-detection effort, EMER·
`ALD, a distributed scalable tool suite for tracking mali(cid:173)
`cious activity through and across large net.works [20].
`EMERALD iuiro<luces a building-block approach to
`nP-t>vork ,o;urveillance, attac:k isolation, and a.utomated
`response. The a.pproa.ch employs highly distributed, in(cid:173)
`dependently tunable, surveillance and response mon(cid:173)
`itors that arc deployable puly111orphically at various
`ahstrar.t laye:rs in a large network. These monitors
`den1onstrate a streamlined intrusion-detect.ion design
`that co1nbines signature analysis with statistical proftl(cid:173)
`ing to provide localized real-t.ime protect.ion of the most
`widely used network services and components on the
`Internet.
`
`Among the general types of analysis targets that
`EMERALD monitors arc network gateways. We de-
`8cribe several analysis t.el".hniqnes that EMERALD im(cid:173)
`plements, and discuss their use in analyzing malicious,
`faulty, and other exceptional uct\vork activity. F,MF,R(cid:173)
`ALD's ~urveillance rnodules will monitor entry points
`that separate external network traffic fro1n an enterprise
`network and its constituent local <lornain,;. 2 We present
`thc~e 8urveillance techniques as complementary to the
`filtering mechanisms of a large enterprise network, and
`illustrate their utility in directly cuhancing the ,o;ec:urity
`and stability of nctV1·urk operations.
`
`"\Ve first consider the candidate event streams that
`pass through network entry points. Critical to the ef(cid:173)
`fective 111011itoring of operations is the careful selection
`an<l organization of these event strca1118 such that an
`analysis bWled 011 a selected event stream will provide
`111ea11i11gful insight into the target activity. We identify
`effective analytical techniques for processing the event
`stream given specific analysis objectives. Sections 4
`ancl 5 explore how both staListical anomaly detection
`and signature analysis can be applied to iclentify activ(cid:173)
`ity worthy of review and µossible re,o;ponse. All such
`
`gained wide deployment in some Departn1ent of n~reuse network
`facilities.
`2 We use the terms enterprise and intr11'<"1 inLerchnngcably;
`both exi5t ultin1al.ely as cooperative conunun.ities of incl.,p•m(cid:173)
`d .. utly administered domains, com1nunirat.iug 1.ogethcr with sup(cid:173)
`portive network i11fr...,.tructure such as firewalls, router"· and
`lirirlges.
`
`In procee<lillg~ u! Lhe 199!i !SOC' S"ymposi111n on N~twork and Distribu.ted Sy.1tfln1s Secur;ly
`
`2 of 13
`
`Commerce Bancshares, Inc., et al.
`Exhibit 1004
`Page 2
`
`

`

`clai111s are suµµurtt.:d by exan1ples. More broadly, in
`Section fl \Ve discuss the correlation of analysis results
`producf'rl hy surveillance cornµunents deployed indepen
`dently throughout the entry points of our protected in(cid:173)
`tranct. We discuss how events of limitecl significance to
`a local surveillance monitor may be aggregated \Vith re(cid:173)
`sults frorn other ::;trategically deployed monitors to pro(cid:173)
`vide insight into more \vide-scale proble111::; or threats
`against the intranet. Section 7 discusses the issue of
`response.
`
`3 Event Stream Selection
`
`The success or failure of event analy1:1i1:1 ::;hould be quanti(cid:173)
`tatively measured for q11 ::i.litie.~ such as acc1irar,y and per(cid:173)
`forn1ance: both are assessable through testing. A more
`difficult but equally i1nportant n1etric to assess is con1-
`pleteness.
`'VVith regard to net¥lork surveillance, inac(cid:173)
`curacy is reflected in the number of legitimate transac(cid:173)
`tion::i flagged as abnormal or malicious (false positives),
`inromph~teness is reflected in Lhe 11u111ber of har111ful
`transactions that escape detection (falsr. negatives), and
`perforn1ance is 111easured by the rate at which transac(cid:173)
`tions can be processed. All three nleasurements of suc(cid:173)
`cess or failure directly depend on the quality of the event
`stream upon which the analysis is bll!ed. Here, we con(cid:173)
`sider the objective of providing real-time surveillance
`of TCP /IP-based networks for 111alicious or exceptional
`network traffic. In parlicular, our net\\'ork surveillance
`mec,hanisms ran he integrated onto, or interconnectecl
`with, network gateways that filter traffic between a pro·
`tected intranet and external networks.
`
`IP traffic represents an interesting candidate event
`stream for analysis.
`Individually, packets represent
`parsablc activity records, where key data \viLhin Lhe
`header and data segment can he statistically analyzed
`and/or heuristically parsed for response-worthy activity.
`llowever, the sheer volume of potential packets dictates
`careful llilsess111ent of ways to optirnally organizr. pack(cid:173)
`ets into streams for efficient parsing. Thorough filtering
`of events and event fields such that the target activ(cid:173)
`ity is concisely isolated, should be applied early in the
`processing stage to reduce re::;ource utilization.
`
`\l\'ith respect to T"CP /IP gateway tr attic monitoring,
`we have investigated a variety of ¥1ays to categorize
`and isolate groups of packets fron1 an arbitrary packt.:t
`stream. Individual packet strean1s can be filtered bll!ed
`on different isolation criteria, ::iuch fill
`
`• Pas.~-t!tro'ugh lru.f]ic. packets allo\ved into the inter(cid:173)
`nal network frorn exlernal sources.
`
`• Protocol-spec({ic lru_{fic: µackets pertaining to a
`common protocol
`fl..~ designated in Lhe packet
`header. One example is the stream of all ICMP
`packets that reach the gateway.
`
`•
`
`l!nassigned port lrajfic: packets targeting ports to
`which the administrator has not assigned any net(cid:173)
`work service and that also ren1ain unblocked by the
`firewall.
`
`• Tran11porl 1nunagcu1enl rnes::;o._qc::;: packets involv(cid:173)
`ing t.ra.nsport-111.yer <'.onnection establishment, con(cid:173)
`trol, and termination (e.g., TCP SYN, RESET,
`ACK, <window resize>).
`
`• Source-address 1nonitor1ng: packets v.·hose source
`addresses match well-known external sites (e.g.,
`connections fron1 satellite offices) or have raised
`::;uspicion fro111 other n1onitori11g efforts.
`
`• Deslinaliori-add1·ess nioniluring: all packets whose
`clestination addresses match a given internal host
`or workstation
`
`• Application-layer monitoring: packets targeting a
`particular network service or application. 1'his
`strean1 isolation 1nay translate to parsing packel
`headers for TP/port matches (assuming an estab~
`lished binding between port and service) and re~
`building datagrarns.
`
`lu the following sections we discuss how such t.raffi<'.
`st,reams can he statistic::i.\ly and heuristically analyzed
`to provide insight into malicious and erroneous external
`traffic. Alternative sources of event data are also avail(cid:173)
`able fro1n the report logs produced hy the various gate(cid:173)
`ways, -firewalls, routers, and proxy-servers (e.g., router
`syslogs can in fact be used to collect packet inforn1a(cid:173)
`tion from several products). We explore how stlltistical
`and signature analysis techniques can be employed to
`monitor various elen1ents within 'J'CP /IP event strean11:1
`that flow through network gateways. '\Ve present spe(cid:173)
`cific techniques for detecting external entities that at(cid:173)
`Le111pl lo subvert or bypru;s internal network services.
`Techniques are suggested for detecting attacks against
`the underlying network infrastructure, including attacks
`using corruption or forgery of legitin1ate traffic in an at(cid:173)
`tempt to negatively affect routing services, aµplication(cid:173)
`Jayer services, or other uct;vork conlrols.
`\Ve suggest
`
`• Discarded traffic: packets not allowed through the
`gateway becau::;e they violate filtering rules. 3
`30f par~icular added value in assessing this traffic would be
`
`some indication of why a given pack"t WM ,-.,j~cted. A generic
`Rolut.inn fur deriving this di•po•ition information without depen(cid:173)
`dencies on the firewall or router is difficuh.. Suo:l1 infurn1.atjon
`would be a. useful en\1a11ct<nLenl lu pm::kct"rcjection handlers.
`
`In proceP.dings of the 1998 JSOC Sy1npo~iu1r1 011 iVetwork and Distributed Sy~tF.ms Secui·ity
`
`3 of 13
`
`Commerce Bancshares, Inc., et al.
`Exhibit 1004
`Page 3
`
`

`

`how to extend onr s11rveillance Lechuique:; to recognize
`network faults and other exceptional acLivity. \Ve also
`<liscuss issues of distributed result correlation.
`
`4 Traffic Analysis with Statisti(cid:173)
`cal Anomaly Detection
`
`SRI has been involved in statistical anomaly-detection
`research for over a decade [1, 5, 10]. Our previous "'ork
`focused on the profiling of user activity through audit(cid:173)
`trail analysis. \.Vithin Lhe EMER.ALD µrojcct, we arc
`extending the underlying statistical algorith mi; to profile
`various aspects of network traffic in search of response(cid:173)
`or alP:rt-worthy anornalie;;.
`
`'!'he statistical subsystem tracks subject activity via
`one or more variables called 1neasures. '!'he statistical
`algorilJuus e1uploy four clilllses of 1neW>urcs; categorical,
`continnoui;, intensity, and event distribution. Categori(cid:173)
`cal me::i;;11res are those that a;;;sume values from a cate(cid:173)
`gorical set, such as originating host identity, destination
`host, and port number. G'ontinuous 1neasures are those
`for which observed values are uu1ueric or ordinal, such
`as number of bytes transferred. Derived measures also
`track the intensity of activity (that is, the rate of events
`per unit time) and the "n1eta-distribution" of the n1ea(cid:173)
`sures affected by recent events. These derived measure
`types a.re referred to as intensity and event distribution.
`
`The system we have developed maintains and up(cid:173)
`dates a description of a subject's behavior with respect
`to these mea.9urc types in a co111pact, efficieuLly updated
`profile. The profile is subdivided into i;hort- an<l long(cid:173)
`term elements. The short-term profile accumulates val(cid:173)
`ues between updates, and exponentially ages values for
`comparison to the long-Ler111 profile. As a r.onsequenr.e
`of t,he aging mPrhanism, the short-term profile char(cid:173)
`acterizes the recent activity of the subject, where "re(cid:173)
`cent" is detern1ined by the dyna111ically configurable ag(cid:173)
`ing paran1eters used. At update tin1e (typically, a time
`of low systc111 acLiviLy), Lhe update funr,tion folds the
`shorL·Lerrn values obi;erved since the la.5t update into the
`long-term profile, and the short-tern1 profile is cleared.
`'fhe long-tern1 profik: is it<iclf slowly aged to adapt to
`changes in subJect activity, A1101ua!y scoring cornpares
`related attributes iu the short-terrn profile against the
`long-ier111 profile. Ai; all evaluations are done against
`c111pirical distrib11tionR, no assumptions of paran1etric
`distributions arf' made, and n1ulti-n1odal and categori(cid:173)
`cal distributions are accommodated. l 1'urthermore, the
`algorithms we have developed require no a priori knowl(cid:173)
`edge of intrusive or exceptional activity. A n1ore de(cid:173)
`tailed mathematical description of these algoriLhnIB is
`
`given in [9, 26].
`Our earlier work r.onsidered the subject class of users
`of a con1puter system and the corresponding event
`::;trean1 the systcn1 audit trail generated by user ac(cid:173)
`tivity. Within the EMER,ALD project, '?l'e geueralize
`these concepts so that components and software such
`a.9 network gatev;ays, proxies, and network services can
`the1nselves be 111ade subject cla:;scs. The generated
`event streams are obtained from log files, packeL anal(cid:173)
`ysis, and-where required-special-purpose instrumen(cid:173)
`tation made for services of interest (e.g., F'l'P, H'l"l'P,
`or SMTP). As appropriaLe, aIJ event stn::a111111ay be an(cid:173)
`alyze<l a.<; a single subject, or as rnultiple subjects, and
`the san1e network activity can be analyzed in several
`ways. For exan1ple, an event strcan1 of dropped pack(cid:173)
`ets permits analyses that trar.k the rea.<;on each par,ket
`was rejected. Under such a scenario, the firewall re(cid:173)
`jecting the packet is the subject, and the measures of
`interesL are Lhe re!IBon the packeL was dropped (a cat(cid:173)
`egorical measure), and the rate of <lropped par.kets in
`the recent past (one or more intensity measures tuned
`to time intervals of seconds to 1ninutes). Alternatively,
`these dropped µackets 111ay be parse<l in fiIJer detail,
`supporting other ::i.n::i.lyses where the suhject is, for ex(cid:173)
`an1ple, the identity of the originating host.
`
`EI\lERALU can also choose to separately define
`satelliLe offices and "rest of world" as different subjecti;
`for the same event stream. That is, we expect distinc(cid:173)
`tions from the satellite office's use of services and ac(cid:173)
`cess to assets to deviate "'idely from sessions originatiIJg
`from external nouaffiliated siLes. Through satellite sei;(cid:173)
`sion profiling, E\fERA T. n r.an monitor traffic for signs
`of unusual activity. In the case of the F'l'P service, for
`exan1ple, each user who gives a login 11a111e is a subject,
`and "anony1nous11 is a subjecL as well. Another exam(cid:173)
`ple of a subject is the network gateway itself, in which
`case there is only one subject. All subjects for the sa1ne
`cvenL strea111 (LhaL is, all subjer.ti; '-"'ithin a subject class)
`have the i;::i.me measures defined in their profiles, but the
`internal profile values are different.
`
`Ai; WP: migrate our statistical algorithms that had pre(cid:173)
`viously focused on user audit trails with users as sub(cid:173)
`jects, '-"'e generalize our ability Lo build more abstract
`profiles for varie<l types of activity captured within our
`generalized notion of an event stream. In the context
`of statistically analyzing TCP /IP traffic strea1ns, profil(cid:173)
`ing can be derived fron1 a variety of traffic perspectivei;,
`including profiles of
`
`• Protocol-sper.ific transactions (e.g., all ICMP ex(cid:173)
`r.hftnges)
`
`• Sei;sions between specific internal hosts au<l/or spe-
`
`In pro<.:ccdings of the 1998 !SOC Symposium nn i'>iP.twork <Hltl Di~l•·ibult'd Sy8tems Security
`
`1 of lJ
`
`Commerce Bancshares, Inc., et al.
`Exhibit 1004
`Page 4
`
`

`

`cific external sites
`
`• Application-layer-specific sessions (e.g., anonymous
`FTP sessio11s profiled individually and/or collec(cid:173)
`tively)
`
`• Discarded traffic, measuring a.ttrih11te;s such as vol(cid:173)
`uH1e and disposition of rejections
`
`• Connection requests, errors, and unfiltered trans(cid:173)
`mission rates and disposition
`
`Event records are generated either as a result of activ(cid:173)
`ity or at periodic intervals. 111 our case, activity records
`are based on the content of IP packets or transport(cid:173)
`layer data.gr~.m.~. 011r event filters also construct. inter(cid:173)
`val sun1111ary records, which contain acc.umulated net(cid:173)
`work traffic statistics (at a 1ninin1um, number of packets
`and 11u111bcr of kilobytes transferred). 'l'hese records are
`constructed at the end of each interval (e.g., once per N
`seconds).
`
`F.MF.RALD's statistical algoritl11n adjusts its short(cid:173)
`tern1 profile for the measure values observed on the
`event record. The distribution of recently observed val(cid:173)
`ues is evaluated against the long-ter111 profile, and a
`distance between the t"''o is obtained. The difference
`is compared to a historically adaptive, subject-specific
`deviation. The empirical distribution of this deviation
`is Lransforrned to obtain a score for the event. Ano111a(cid:173)
`lous events are thosP whose sr.ores exr.eed a historically
`adaptive, subject-specific score threshold based on the
`'l'his nonparametric ap(cid:173)
`en1pirical score distribution.
`proach ha11dles all rneasure types a11d rnakes 110 !l.'lsurnp(cid:173)
`tions on t.he modality of the difitribution for c.ontinuous
`measures.
`
`The; following se;r.tions provide example scenarios of
`exceptional network activity that can be measured by
`an EMEllALD statistical engine deployed to network
`gateways.
`
`4.1 Categorical Measures 111 Network
`Traffic
`
`Categorir.al measures assume values fron1 a discrete,
`nonordered set. of possibilities. Exan1ples of categori(cid:173)
`c;:i.1 measures include
`
`• Snurce/destination address: One expects, for ex(cid:173)
`ample, accesses from satellite oftict.:s Lu originate
`from a set of known host identities.
`
`• Command issued: While any single couunand may
`not in itself be a1101nalous, >iome intri1sion scenar(cid:173)
`ios (such as "doorknoh rattling") give rise to an
`
`unusual 1nix of commands in the short-term pro(cid:173)
`ftlc.
`
`• Protocol: As ,.,·ith r.orr1rr1ands, a single retp1cst of
`a given protocol may not be anomalous, bnt an
`unusual 1nix of protocol requests, reflected in the
`Hhort-ter1n profile, 1nay indicate an intrusion,
`
`• Errors and privilege violat.io11s: We track the return
`code from a command as a categoric.a.I me;:isnre; we;
`expect the distribution to reflect only a small per
`r.e;nt of abnorrnal returns (the actual rate is learned
`in the long-te;rm profile). While some rate of errors
`is normal, a high number of exceptions in the recent
`past is abnormal. This is reflected both in unusual
`freq11enr.ie;s for abnormal categories, deter.t.ed here,
`and unusual count of abnorn1al returns, tracked as
`a continuous measure as described in Section 4.2.
`
`• Malforn1ed service requests: Categorical measures
`r.an track the occurrence of various for1ns of bad
`requests or malformed packets directed to a specific
`network service.
`
`• ~lalforrned packet disposition: Packets are dropped
`by a packet filter for a variety of reasons, many of
`winch are innocuous (for exan1ple, badly formed
`packet header). U11usual patterns of packet rejec(cid:173)
`tion or e;rror messages could lead to insight into
`problems in neighboring systems or more serious
`atten1pts by external sites to probe internal assets.
`
`• File handles: Certain subjects (for exarnple, anony(cid:173)
`mous FTP users) are restricted as to which files
`they can access. Atten1pts to access other tiles or to
`1vritc rcad-011ly files appear anornalous Snr.h events
`l:lre ofteu detectable by Hignature analysis AA well.
`
`The statistical componi>nt builds empirical distribu(cid:173)
`tions of the c.ategory values encountered, even if the list
`of possible values is open-ended, and has n1echanisn1s
`for "aging out" categories whose long-terrn probabili(cid:173)
`ties drop below a threshold.
`
`The following is an example of categorical measures
`used in the surveillance of proxies for services such
`as SMTP or FTP. Cuusidcr a typical data-exchange
`sequence between an externul clic11L aud an inter(cid:173)
`nal server within the protected network. Anonymous
`.F'f P is restricted to certain files and direr.tories; the
`na111cs of these are categories for measures pertaining
`to file/direr.tory reads and (if permitted) writes. At(cid:173)
`ternpted acr,esses to unusual directories appear anoma(cid:173)
`lous. i\1onitors dedicated to ports include a categorical
`n1easure whose values arc the protocol used. Invalid re(cid:173)
`quests oft<.:n lead to an acce>is violation error; the type
`
`In proceedings of the 1998 JSOC SympcMium 011 A'etwork t111tl Di~lributt'd Syaterns Security
`
`5 of 13
`
`Commerce Bancshares, Inc., et al.
`Exhibit 1004
`Page 5
`
`

`

`of error associated with a request is a11ot 11er ex1u11ple of
`a categorical n1easure, and the count or rate of errors
`in the rt:cent past is tracked as continuous measures, as
`dPsc:rihed in Secliou 4.2.
`
`1.2 Conti11uous Measures in Network
`Traffic
`
`Continuous measures assun1e values from a continuous
`or ordinal set. Exa111ples include inter-event ti1ne ( dif(cid:173)
`ference in time stamps bet;veen couseculive eveuts fru111
`the same stream), counting mea.sures snc:h as thf' nun1-
`ber of errors of a particular type observed in the recent
`pasL and network traffic uH.:asures (nun1ber of packets
`and number of kilobytes). The slalislical subsystc111
`treats contin11011s mefl..~ures by firsl allocaling bins ap(cid:173)
`propriate to t.he range of values of the underlying mea(cid:173)
`sure, and then tracking the frequency of observation of
`each value range. In Lhis 'Nay, rnulli-111odal distributions
`are acc.ommodated and muc.h of the computational ma(cid:173)
`chinery used for categorical measures is shared.
`
`Continuous n1e<l.'lures are useful not only fur iutrusiou
`deter.tion, hnt also support the monitoring of health and
`status of the net\.vork from the perspective of connectiv(cid:173)
`ity and throughput. An instantaneous measure of traffic
`volun1e maintained by a gateway n1onitor can detect a
`sud<len a11<l unexpected loss in the <lala rate of received
`packets, when this volume falls outside historic.al norms
`for the gateway. 'l'his sudden drop is specific both to
`Lhe galeway (the subjecl, in this case) an<l Lo Lhe time
`of <lay (e.g., Lhe average sustained traffic rate for a rna.(cid:173)
`jor network artery is much different at 11:00 a.111. than
`at n1idnight).
`
`In our exarnple discussion of an FTP service in Sec(cid:173)
`tion 4.1, atte111pts to access unallowed directories or filris
`result in errors. The recAntly ohsArved ra.te of such er(cid:173)
`rors is continuously compared with the rate observed
`over sin1ilar time spans for other FTP sessions. So111e
`101-v rate of error due t.o rnisspAllings or innocent at-
`1e1npts is to he rixpAc:ted, and this would be reflected
`in the historical profile for these ineasures. An excess
`beyond historical norms indicates a110111alous activity.
`
`Continuous rucasures can also work in conjunction
`with categorical measures to detecl excessive data trans(cid:173)
`fers or file uploads, or cxct:::.:oive 111ail relaying, fl..~ well a.<;
`excessive service-layer errors by external clients. Cate(cid:173)
`gorical a1Hl conlinuous measures have proven to be the
`111osl useful for anomaly detection in a variety of con(cid:173)
`tPxts.
`
`\Ve next describe the t>vo <lerive<l 1neasure types, in(cid:173)
`tensity and cvcrtl di:>tr·ibuliun, vvhirh deter.t annmalies
`
`related to recent traffic volume and the mix of measurPs
`afTecte<l by this traffic.
`
`4.3 Measuri11g Network Traffic Inten(cid:173)
`sity
`
`lntensity measures distinguish whether a given volume
`of traffic aµpears consistent with historical observations.
`ThesP mPasnres reflpct the int.ensily of Lhe evenl strea111
`(nun1ber of events per unit time) over time intervals that
`are Lunablc. Tyµically, we have defined three intensity
`measures per profile, which, \\•ilh respect to user aclivit.y
`monitoring, were scaled at intervals of 60 seconds, 600
`seconds, and 1 hour.
`..\.pplied to raw event strean1s,
`inlensity ine<l.'lures art: particularly suited for <lctcctiug
`flooding attacks, while also providing insight. inlo olher
`anomalies.
`
`E:\1ERALD uses volnme ana.lyses to help detect the
`introduction of malicious traffic, such as traffic intended
`to cause service denials or perforn1 intelligence gath(cid:173)
`Pring, whPrP such traffic m11.y not nec.Pssarily he vio(cid:173)
`lating filtering policies. A sharp increase in the over(cid:173)
`all volun1t: of di:ocar<le<l packets, us well as analysis of
`the disposition of the disr,arded packets (fl..~ disr,11ssrid
`in Section 4.1 ), can provide insight into 11nintentionally
`malformed packets resulting from poor line quality or
`internal errors in neighboring hosts. High volumes of
`discarded packels can also indicate rnore maliciously in(cid:173)
`tended transmissions such as scanning of UPD ports or
`IP address scanning via ICMP echoes. Excessive num(cid:173)
`bers of 111ail expansio11 requests (EXPN) may indir.ate in(cid:173)
`telligence gathering, perhaps by spammers. 'fhese and
`other application-layer forms of doorknob raitli11g can
`Uc <lcit:cic<l Uy an £.1\ifERALD statistical enginP ¥1hen
`filtering is not desired.
`
`Allcrnalively, a sharp increa.~e in events viewed
`across longer durations may prnvide insight into a con(cid:173)
`sistent P:ffort t.o limit or prevent successful traffic flow.
`Intensity measures of transport-layer conneclion re(cid:173)
`quests, such as a volume analysis of SYN-RST mes(cid:173)
`sages, could indicate the occurrence of a SYN-attack [17]
`against port availability (or possibly for port scanning).
`Variants of this could includA intensity measures of
`TCP/FIN iucssa,l!;es [14], considered a more ;;tealthy
`forn1 of port scauning.
`
`Monitoring overall traffic volun1c an<l bursty events
`by using both intensity and couti11uous 1nea.~ures pro(cid:173)
`vides son1e interesting advantrtges over other monitoring
`aµproaches, such as user-definable heuristic rules that
`specify fixPd thresholds. In particular, the intensity of
`events over a duration is relative iu the sense that the
`term "high volume" n1ay reasonably be considered dif-
`
`In prorf'P<iings of the 1998 lSOC Syr11posiu.m on Jl.'etwork (lnd /)i~fributed Systenos Security
`
`6 of 13
`
`Commerce Bancshares, Inc., et al.
`Exhibit 1004
`Page 6
`
`

`

`ferent ut 1nidnight than at 11:00 a.111. The notion of
`high bursts of events might similarly be unique to the
`role oft11e target systen1 in the intrane