Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 2 of 16 PageID #: 3399
`
`1 ~ .. 'h - . !!
`
`iLJl ~::~ ..
`
`DOCKET NO: 149-0102US
`
`APPLICATION
`
`FOR
`
`UNITED STATES LETTERS PATENT
`
`TITLE: Two-Phase Root Cause Analysis
`
`INVENTORS: Michael R. Warpenburg and Michael J. Scholtes
`
`Express Mail No: EV195559360US
`
`Date: April 22, 2003
`
`Prep<tred by: WONG, CABELLO, LVTSCH, RUTHERFORD & BRVCCVLERI, P.C.
`HOUSTON, TEXAS
`
`(VOICE) 832-446-2400
`
`(FACSIMILE) 832-446-2424
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 3 of 16 PageID #: 3400
`
`TWO-PHASE ROOT CAUSE ANALYSIS
`
`DOCKET NO: 149-0102US
`
`Background
`[0001] The invention relates generally to the field of event detection and fault
`
`5
`
`diagnosis for computer systems and, more particularly but not by way of limitation, to
`
`techniques (devices and methods) for defining and using fault models for the
`
`monitoring, diagnosis and recovery of error conditions in a enterprise computing
`
`system.
`
`10
`
`[0002] Contemporary corporate computer networks comprise a plurality of different
`
`computer platforms and software applications interconnected through a number of
`
`different paths and various hardware devices such as routers, gateways and switches.
`
`Illustrative computer platforms include desktop personal computers, engineering
`
`workstations, dedicated file, application and mail servers and mainframe computer
`
`15
`
`systems. Illustrative software applications include accounting, payroll, order entry,
`
`inventory, shipping and database applications. The collection of such entities- hardware
`
`and software- is often referred to as an "enterprise."
`
`[0003] As enterprises have become larger and more complex, their reliability has
`
`20
`
`become ever more dependent upon the successful detection and management of
`
`problems that arise during their operation. Problems can include hardware and software
`
`failures, hardware and software configuration mismatches and performance
`
`degradation due to limited resources, external attacks and/or loss of redundancy.
`
`Operational problems generate observable events, and these events can be monitored,
`
`25
`
`detected, reported, analyzed and acted upon by humans or by programs. It has been
`
`observed that as an enterprise grows (i.e., incorporates more monitored components(cid:173)
`
`hardware and software), the rate at which observable events occur increases
`
`dramatically. (Some studies indicate event generation rates increase exponentially with
`
`enterprise size.) Quickly and decisively identifying the cause of any given problem can
`
`30
`
`be further complicated because of the large number of sympathetic events that may be
`
`- 1 -
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 4 of 16 PageID #: 3401
`
`DOCKET NO: 149-0102US
`
`What is claimed is:
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`s
`
`g
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`1
`
`2
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`1.
`
`An enterprise fault analysis method, wherein at least a portion of the
`
`enterprise is represented by a enterprise-specific fault model having a plurality of
`
`nodes, comprising:
`
`receiving an event notification for a first node in the fault model;
`
`performing an up-stream analysis of the fault model beginning at the first node;
`
`identifying a second node, the second node having a status value modified
`
`during the up-stream analysis to indicate a failed status;
`
`performing a down-stream analysis of the fault model beginning at the second
`
`node;
`
`identifying those nodes in a contiguous path between the second node and the
`
`first node in the fault model whose impact values indicate an impacted performance
`
`condition in accordance with the down-stream analysis;
`
`reporting the second node as a root cause of the received event notification; and
`
`reporting at least one of the identified nodes as impacted by the root cause of
`
`the received event notification and not as root causes of the received event notification.
`
`2.
`
`The method of claim 1, wherein the enterprise-specific fault model comprises
`
`an Impact Graph.
`
`3.
`
`The method of claim 1, wherein the act of performing an up-stream analysis
`
`comprises:
`
`evaluating an inference policy associated with the first node and setting a status
`
`value associated with the first node in accordance therewith; and
`
`evaluating inference policies associated with up-stream nodes to the first node
`
`and setting a status value associated with each evaluated up-stream node in
`
`accordance therewith.
`
`-20-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 5 of 16 PageID #: 3402
`
`DOCKET NO: 149-0102US
`
`4.
`
`The method of claim 3, wherein the act of evaluating inference policies is
`
`terminated when no up-stream nodes from the last evaluated node exist.
`
`5.
`
`The method of claim 4, wherein the act of evaluating inference policies is
`
`further terminated when a status value associated with a node does not change based
`
`on evaluation of an inference policy associated with the node.
`
`6.
`
`The method of claim 4, wherein the act of evaluating inference policies is
`
`further terminated when a status value associated with a node is a measured status
`
`value.
`
`The method of claim 1, wherein the act of identifying a second node further
`
`7.
`comprises identifying one or more nodes that are most up-stream from the first node.
`
`8.
`
`The method of claim 7 further comprising identifying, as the second node,
`
`an arbitrary one of the one or more identified nodes.
`
`The method of claim 3, wherein the status value associated with a node
`
`9.
`comprises a Boolean value.
`
`10.
`
`The method of claim 3, wherein the status value associated with a node
`
`comprises a real-number value.
`
`11.
`
`The method of claim 3, wherein a status value associated with a node
`
`further has one or more associated attributes.
`
`12.
`
`The method of claim 11, wherein one of the one or more associated
`
`attributes comprises a temporal attribute.
`
`1
`
`2
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`- 21 -
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 6 of 16 PageID #: 3403
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`4
`
`s
`
`6
`
`7
`
`1
`
`2
`
`1
`
`2
`
`3
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`DOCKET NO: 149-0102US
`
`13.
`
`The method of claim 11, wherein one of the one or more associated
`
`attributes comprises an indication to identify the status value as being a measured
`
`value or an inferred value.
`
`14.
`
`The method of claim 1, wherein the act of performing a down-stream
`
`analysis comprises:
`
`evaluating an impact policy associated with the second node and setting an
`
`impact value associated with the second node in accordance therewith; and
`
`evaluating impact policies associated with down-stream nodes to the second
`
`node and setting an impact value associated with each evaluated down-stream node in
`
`accordance therewith.
`
`15.
`
`The method of claim 14, wherein the act of evaluating impact policies is
`
`terminated when no down-stream nodes from the last evaluated node exist.
`
`16.
`
`The method of claim 15, wherein the act of evaluating impact policies is
`
`further terminated when an impact value associated with a node does not change
`
`based on evaluation of an impact policy associated with the node.
`
`17.
`
`The method of claim 14, wherein the impact value associated with a node
`
`comprises a Boolean value.
`
`18.
`
`The method of claim 14, wherein the impact value associated with a node
`
`comprises a real-number value.
`
`19.
`
`The method of claim 14, wherein an impact value associated with a node
`
`further has one or more associated attributes.
`
`20.
`
`The method of claim 19, wherein one of the one or more associated
`
`attributes comprises a temporal attribute.
`
`- 22-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 7 of 16 PageID #: 3404
`
`DOCKET NO: 149-0102US
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`s
`
`9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`21.
`
`The method of claim 1, wherein the act of reporting the second node as a
`
`root cause comprises visually displaying an alarm condition for said second node to a
`
`user.
`
`22.
`
`The method of claim 1, wherein the act of reporting at least some of the
`
`identified nodes as impacted by the root cause comprises visually identifying the at
`
`least one of the identified nodes differently from the second node.
`
`23.
`
`The method of 22, further comprising filtering event notifications received by
`
`at least one of the identified nodes so as to not report said event notification to a user
`
`as a root cause failure.
`
`24.
`
`A program storage device, readable by a programmable control device,
`
`comprising instructions stored on the program storage device for causing the
`
`programmable control device to:
`
`receive an event notification from a first node, said first node one of a plurality of
`
`nodes in an enterprise-specific fault model;
`
`perform an up-stream analysis of the fault model beginning at the first node;
`
`identify a second node, the second node having a status value modified during
`
`the up-stream analysis to indicate a failed status;
`
`perform a down-stream analysis of the fault model beginning at the second
`
`node;
`
`identify those nodes in a contiguous path between the second node and the first
`
`node in the fault model whose impact values indicate an impacted performance
`
`condition in accordance with the down-stream analysis;
`
`report the second node as a root cause of the received event notification; and
`
`report at least one of the identified nodes as impacted by the root cause of the
`
`received event notification and not as root causes of the received event notification.
`
`- 23-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 8 of 16 PageID #: 3405
`
`DOCKET NO: 149-0102US
`
`1
`
`25.
`
`The program storage device of claim 24, wherein the enterprise-specific fault
`
`2 model comprises an Impact Graph.
`
`1
`
`2
`
`3
`
`4
`
`s
`
`6
`
`7
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`4
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`1
`
`2
`
`26.
`
`The program storage device of claim 24, wherein the instructions to perform
`
`an up-stream analysis comprise instructions to:
`
`evaluate an inference policy associated with the first node and set a status value
`
`associated with the first node in accordance therewith; and
`
`evaluate inference policies associated with up-stream nodes to the first node and
`
`set a status value associated with each evaluated up-stream node in accordance
`
`therewith.
`
`27.
`
`The program storage device of claim 26, wherein the instructions to evaluate
`
`inference policies stop evaluating up-stream nodes when no up-stream node from the
`
`last evaluated node exists.
`
`28.
`
`The program storage device of claim 27, wherein the instructions to evaluate
`
`inference policies stop evaluating up-stream nodes when a status value associated with
`
`a node does not change based on evaluation of an inference policy associated with the
`
`node.
`
`29.
`
`The program storage device of claim 27, wherein the instructions to evaluate
`
`inference policies stop evaluating up-stream nodes when a status value associated with
`
`a node is a measured status value.
`
`30.
`
`The program storage device of claim 24, wherein the instructions to identify a
`
`second node further comprise instructions to identify one or more nodes that are most
`
`up-stream from the first node.
`
`31.
`
`The program storage device of claim 30 further comprising instructions to
`
`identify, as the second node, an arbitrary one of the one or more identified nodes.
`
`-24-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 9 of 16 PageID #: 3406
`
`DOCKET NO: 149-0102US
`
`1
`
`32.
`
`The program storage device of claim 26, wherein the status value associated
`
`2 with a node comprises a Boolean value.
`
`1
`
`33.
`
`The program storage device of claim 26, wherein the status value associated
`
`2 with a node comprises a real-number value.
`
`1
`
`34.
`
`The program storage device of claim 26, wherein a status value associated
`
`2 with a node further has one or more associated attributes.
`
`1
`
`2
`
`1
`
`2
`
`35.
`
`The program storage device of claim 34, wherein one of the one or more
`
`associated attributes comprises a temporal attribute.
`
`36.
`
`The program storage device of claim 34, wherein one of the one or more
`
`associated attributes comprises an indication to identify the status value as being a
`
`3 measured value or an inferred value.
`
`1
`
`2
`
`3
`
`4
`
`s
`
`6
`
`7
`
`1
`
`2
`
`3
`
`37.
`
`The program storage device of claim 24, wherein the instructions to perform
`
`a down-stream analysis comprise instructions to:
`
`evaluate an impact policy associated with the second node and set an impact
`
`value associated with the second node in accordance therewith; and
`
`evaluate impact policies associated with down-stream nodes to the second node
`
`and set an impact value associated with each evaluated down-stream node in
`
`accordance therewith.
`
`38.
`
`The program storage device of claim 37, wherein the instructions to evaluate
`
`impact policies stop evaluating when no down-stream nodes from the last evaluated
`
`node exists.
`
`- 25-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 10 of 16 PageID #: 3407
`
`DOCKET NO: 149-0102US
`
`1
`
`z
`
`3
`
`39.
`
`The program storage device of claim 38, wherein the instructions to evaluate
`
`impact policies is further terminated when an impact value associated with a node does
`
`not change based on evaluation of an impact policy associated with the node.
`
`1
`
`40.
`
`The program storage device of claim 37, wherein the impact value associated
`
`2 with a node comprises a Boolean value.
`
`1
`
`41.
`
`The program storage device of claim 37, wherein the impact value associated
`
`2 with a node comprises a real-number value.
`
`1
`
`42.
`
`The program storage device of claim 37, wherein an impact value associated
`
`2 with a node further has one or more associated attributes.
`
`1
`
`2
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`4
`
`1
`
`2
`
`3
`
`43.
`
`The program storage device of claim 421 wherein one of the one or more
`
`associated attributes comprises a temporal attribute.
`
`44.
`
`The program storage device of claim 24, wherein the instructions to report
`
`the second node as a root cause of the received event notification comprise instructions
`
`to visually display an alarm condition for said second node to a user.
`
`45.
`
`The program storage device of claim 241 wherein the instructions to report at
`
`least some of the identified nodes as impacted by the root cause of the received event
`
`notification comprise instructions to visually identify the at least one of the identified
`
`nodes differently from the second node.
`
`46.
`
`The program storage device of claim 45, further comprising instructions to
`
`filter event notifications received by at least one of the identified nodes so as to not
`
`report said event notification to a user as a root cause failure.
`
`- 26-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 11 of 16 PageID #: 3408
`
`DOCKET NO: 149-0102US
`
`47.
`
`An enterprise including a plurality of operatively coupled monitored
`
`components, hereinafter referred to as nodes, comprising:
`
`a first node adapted to generate an event notification message, said first node
`
`one of a plurality of nodes in an enterprise-specific fault model; and
`
`a monitor agent operatively coupled to the first node and adapted to receive the
`
`event notification message, the monitor agent further adapted to:
`
`perform an up-stream analysis of the fault model beginning at the first
`
`node;
`
`identify a second node, the second node having a status value modified
`
`during the up-stream analysis to indicate a failed status;
`
`perform a down-stream analysis of the fault model beginning at the
`
`second node;
`
`identify those nodes in a contiguous path between the second node and
`
`the first node in the fault model whose impact values indicate an
`
`impacted performance condition in accordance with the down-
`
`stream analysis;
`
`report the second node as a root cause of the received event notification;
`
`and
`
`report at least one of the identified nodes as impacted by the root cause
`
`of the received event notification and not as root causes of the
`
`received event notification.
`
`48.
`
`The enterprise of claim 47, wherein operatively coupled monitored
`
`components comprise software applications executing on a computer system.
`
`49.
`
`The enterprise of claim 47, wherein operatively coupled monitored
`
`components comprise hardware devices for facilitating communication between one or
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`8
`
`9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`1
`
`2
`
`1
`
`2
`
`3 more of the operatively coupled monitored components.
`
`- 27-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 12 of 16 PageID #: 3409
`
`DOCKET NO: 149-0102US
`
`1
`
`2
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`4
`
`50.
`
`The enterprise of claim 47, wherein the enterprise-specific fault model
`
`comprises an Impact Graph.
`
`51.
`
`The enterprise of claim 47, wherein the monitor agent is further adapted to,
`
`during said up-stream analysis:
`
`evaluate an inference policy associated with the first node and set a status value
`
`associated with the first node in accordance therewith; and
`
`evaluate inference policies associated with up-stream nodes to the first node and
`
`set a status value associated with each evaluated up-stream node in accordance
`
`therewith.
`
`52.
`
`The enterprise of claim 471 wherein the monitor agent is further adapted to,
`
`during said down-stream analysis:
`
`evaluate an impact policy associated with the second node and set an impact
`
`value associated with the second node in accordance therewith; and
`
`evaluate impact policies associated with down-stream nodes to the second node
`
`and set an impact value associated with each evaluated down-stream node in
`
`accordance therewith.
`
`53.
`
`The enterprise of claim 47, wherein the monitor agent is further adapted to
`
`report the second node as a root cause of the received event notification by visually
`
`displaying an alarm condition for said second node to a user.
`
`54.
`
`The enterprise of claim 47, wherein the monitor agent is further adapted to
`
`report at least some of the identified nodes as impacted by the root cause of the
`
`received event notification comprise instructions to visually identify the at least one of
`
`the identified nodes differently from the second node.
`
`- 28-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 13 of 16 PageID #: 3410
`
`DOCKET NO: 149-0102US
`
`1
`
`2
`
`3
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`s
`
`9
`
`10
`
`u
`
`12
`
`13
`
`14
`
`1
`
`2
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`55.
`
`The enterprise of claim 54, wherein the monitor agent is further adapted to
`
`filter event notifications received by at least one of the identified nodes so as to not
`
`report said event notification to a user as a root cause failure.
`
`56.
`
`A fault analysis method, wherein at least a portion of a system is represented
`
`by a system-specific fault model having a plurality of nodes, comprising:
`
`receiving an event notification for a first node in the fault model;
`
`performing an up-stream analysis of the fault model beginning at the first node;
`
`identifying a· second node, the second node having a status value modified
`
`during the up-stream analysis to indicate a failed status;
`
`performing a down-stream analysis of the fault model beginning at the second
`
`node;
`
`identifying those nodes in a contiguous path between the second node and the
`
`first node in the fault model whose impact values indicate an impacted performance
`
`condition in accordance with the down-stream analysis;
`
`reporting the second node as a root cause· of the received event notification; and
`
`reporting at least one of the identified nodes as impacted by the root cause of
`
`the received event notification and not as root causes of the received event notification.
`
`57.
`
`The method of claim 56, wherein the system-specific fault model comprises
`
`an Impact Graph.
`
`58.
`
`The method of claim 56, wherein the act of performing an up-stream analysis
`
`comprises:
`
`evaluating an inference policy associated with the first node and setting a status
`
`value associated with the first node in accordance therewith; and
`
`evaluating inference policies associated with up-stream nodes to the first node
`
`and setting a status value associated with each evaluated up-stream node in
`
`accordance therewith.
`
`-29-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 14 of 16 PageID #: 3411
`
`DOCKET NO: 149-0102US
`
`59.
`
`The method of claim 58, wherein the act of evaluating inference policies is
`
`terminated when no up-stream nodes from the last evaluated node exist.
`
`60.
`
`The method of claim 59, wherein the act of evaluating inference policies is
`
`further terminated when a status value associated with a node does not change based
`
`on evaluation of an inference policy associated with the node.
`
`61.
`
`The method of claim 59, wherein the act of evaluating inference policies is
`
`further terminated when a status value associated with a node is a measured status
`
`value.
`
`62.
`
`The method of claim 56, wherein the act of identifying a second node further
`
`comprises identifying one or more nodes that are most up-stream from the first node.
`
`63.
`
`The method of claim 62 further comprising identifying, as the second node,
`
`an arbitrary one of the one or more identified nodes.
`
`64.
`
`The method of claim 58, wherein the status value associated with a node
`
`comprises a Boolean value.
`
`65.
`
`The method of claim 58, wherein the status value associated with a node
`
`comprises a real-number value.
`
`66.
`
`The method of claim 58, wherein a status value associated with a node
`
`further has one or more associated attributes.
`
`67.
`
`The method of claim 66, wherein one of the one or more associated
`
`attributes comprises a temporal attribute.
`
`1
`
`2
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`- 30-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 15 of 16 PageID #: 3412
`
`I':'-
`
`..
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`4
`
`s
`
`6
`
`7
`
`1
`
`2
`
`1
`
`2
`
`3
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`DOCKET NO: 149-0102US
`
`68.
`
`The method of claim 66, wherein one of the one or more associated
`
`attributes comprises an indication to identify the status value as being a measured
`
`value or an inferred value.
`
`69.
`
`The method of claim 56, wherein the act of performing a down-stream
`
`analysis comprises:
`
`evaluating an impact policy associated with the second node and setting an
`
`impact value associated with the second node in accordance therewith; and
`
`evaluating impact policies associated with down-stream nodes to the second
`
`node aod setting an impact value associated with each evaluated down-stream node in
`
`accordance therewith.
`
`70.
`
`The method of claim 69, wherein the act of evaluating impact policies is
`
`terminated when no down-stream nodes from the last evaluated node exist.
`
`71.
`
`The method of claim 70, wherein the act of evaluating impact policies is
`
`further terminated when an impact value associated with a node does not change
`
`based on evaluation of an impact policy associated with the node.
`
`72.
`
`The method of claim 69, wherein the impact value associated with a node
`
`comprises a Boolean value.
`
`73.
`
`The method of claim 69, wherein the impact value associated with a node
`
`comprises a real-number value.
`
`74.
`
`The method of claim 69, wherein an impact value associated with a node
`
`further has one or more associated attributes.
`
`75.
`
`The method of claim 74, wherein one of the one or more associated
`
`attributes comprises a temporal attribute.
`
`- 31-
`
`

`
`Case 2:14-cv-00903-JRG Document 99-16 Filed 05/29/15 Page 16 of 16 PageID #: 3413
`
`.::n ..
`
`DOCKET NO: 149-0102US
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`1
`
`2
`
`3
`
`76.
`
`The method of claim 56, wherein the act of reporting the second node as a
`
`root cause comprises visually displaying an alarm condition for said second node to a
`
`user.
`
`77.
`
`The method of claim 56, wherein the act of reporting at least some of the
`
`identified nodes as impacted by the root cause comprises visually identifying the at
`
`least one of the identified nodes differently from the second node.
`
`78.
`
`The method of 77, further comprising filtering event notifications received by
`
`at least one of the identified nodes so as to not report said event notification to a user
`
`as a root cause failure.
`
`- 32-