`
`
`
`V/I HEWLETT®
`
`PACKARD
`
`B3936-90024
`January 1998
`
`© Copyright 1998 Hewlett-Packard Company
`
`DHPN-1004
`Dell Inc. vs. Electronics and Telecommunications, IPR2013-00635
`Page 1 of 260
`
`
`
`Legal Notices
`The information contained in this document is subject to change without
`notice.
`Hewlett-Packard makes no warranty of any kind with regard to this
`manual, including, but not limited to, the implied warranties of
`merchantability and fitness for a particular purpose. Hewlett-Packard
`shall not be liable for errors contained herein or direct, indirect, special,
`incidental or consequential damages in connection with the furnishing,
`performance, or use of this material.
`Copyright © 1995, 1997, 1998 Hewlett-Packard Company.
`This document contains information which is protected by copyright. All
`rights are reserved. Reproduction, adaptation, or translation without
`prior written permission is prohibited, except as allowed under the
`copyright laws.
`Corporate Offices:
`
`Hewlett-Packard Co.
`3000 Hanover St.
`Palo Alto, CA 94304
`Use, duplication or disclosure by the U.S. Government Department of
`Defense is subject to restrictions as set forth in paragraph (b)(3)(ii) of the
`Rights in Technical Data and Software clause in FAR 52.227-7013.
`Rights for non-DOD U.S. Government Departments and Agencies are as
`set forth in FAR 52.227-19(c)(1,2).
`Use of this manual and flexible disc(s), compact disc(s), or tape
`cartridge(s) supplied for this pack is restricted to this product only.
`Additional copies of the programs may be made for security and back-up
`purposes only. Resale of the programs in their present form or with
`alterations, is expressly prohibited.
`A copy of the specific warranty terms applicable to your Hewlett-Packard
`product and replacement parts can be obtained from your local Sales and
`Service Office.
`
`2
`
`
`
`DHPN-1004 / Page 2 of 260
`
`
`
`Contents
`
`1. MC/ServiceGuard at a Glance
`
`What is MC/ServiceGuard?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
`Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
`
`Using this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20
`
`For More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
`
`2. Understanding MC/ServiceGuard Hardware Configurations
`
`Redundancy of Cluster Components . . . . . . . . . . . . . . . . . . . . . . . . . . . .24
`
`Redundant Network Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26
`Redundant Ethernet Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . .26
`Providing Redundant FDDI Connections . . . . . . . . . . . . . . . . . . . . . . .28
`Using Dual Attach FDDI Stations . . . . . . . . . . . . . . . . . . . . . . . . . . . .28
`Fibre Channel Switched Configurations. . . . . . . . . . . . . . . . . . . . . . . .29
`Using a Serial (RS232) Heartbeat Line . . . . . . . . . . . . . . . . . . . . . . . .30
`
`Redundant Disk Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32
`Supported Disk Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32
`Data Protection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
`Monitoring of Disks Through Event Monitoring Service . . . . . . . . . . .34
`Replacement of Failed Disk Mechanisms . . . . . . . . . . . . . . . . . . . . . . .34
`Sample Disk Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34
`Root Disk Limitations on Shared Buses . . . . . . . . . . . . . . . . . . . . . . . .36
`
`Larger Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39
`Active/Standby Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39
`Point to Point Connections to Storage Devices. . . . . . . . . . . . . . . . . . .40
`
`3. Understanding MC/ServiceGuard Software Components
`
`MC/ServiceGuard Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44
`
`How the Cluster Manager Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45
`
`
`
`3
`
`DHPN-1004 / Page 3 of 260
`
`
`
`Contents
`
`Configuration of the Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
`Manual Startup of Entire Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
`Heartbeat Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
`Automatic Cluster Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
`Dynamic Cluster Re-formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
`Cluster Quorum for Re-formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
`Use of the Cluster Lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
`Single Cluster Lock. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
`Dual Cluster Lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
`No Cluster Lock. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
`Backing Up Cluster Lock Information. . . . . . . . . . . . . . . . . . . . . . . . . 50
`
`How the Package Manager Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
`Deciding When and Where to Run and Halt Packages. . . . . . . . . . . . 51
`Starting the Package and Running Application Services. . . . . . . . . . 51
`Service Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
`Using the Event Monitoring Service . . . . . . . . . . . . . . . . . . . . . . . . . . 53
`Using the EMS HA Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
`Stopping the Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
`
`How the Network Manager Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
`Stationary and Relocatable IP Addresses . . . . . . . . . . . . . . . . . . . . . . 55
`Adding and Deleting Relocatable IP Addresses . . . . . . . . . . . . . . . . . 56
`Monitoring LAN Interfaces and Detecting Failure. . . . . . . . . . . . . . . 56
`
`Responses to Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
`Transfer of Control (TOC) When a Node Fails . . . . . . . . . . . . . . . . . . 62
`Responses to Hardware Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
`Responses to Package and Service Failures . . . . . . . . . . . . . . . . . . . . 63
`
`4. Planning and Documenting an HA Cluster
`
`General Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
`Planning for Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
`
`4
`
`
`
`DHPN-1004 / Page 4 of 260
`
`
`
`Contents
`
`Hardware Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69
`SPU Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70
`Network Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70
`Setting SCSI Addresses for the Largest Expected Cluster Size . . . . .72
`Disk I/O Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73
`Hardware Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . .75
`
`Power Supply Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76
`Power Supply Configuration Worksheet. . . . . . . . . . . . . . . . . . . . . . . .77
`
`Volume Group and Physical Volume Planning . . . . . . . . . . . . . . . . . . . .78
`Volume Group and Physical Volume Worksheet . . . . . . . . . . . . . . . . .78
`Volume Group and PV Links Worksheet . . . . . . . . . . . . . . . . . . . . . . .80
`
`Cluster Configuration Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81
`Choosing the Cluster Lock Volume Group . . . . . . . . . . . . . . . . . . . . . .81
`Planning for Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82
`Cluster Manager Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83
`Cluster Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87
`
`Package Configuration Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88
`Logical Volume and Filesystem Planning. . . . . . . . . . . . . . . . . . . . . . .88
`Monitoring Registered Package Resources . . . . . . . . . . . . . . . . . . . . . .89
`Choosing Switching and Failover Behavior . . . . . . . . . . . . . . . . . . . . .90
`Planning for Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92
`Package Configuration File Parameters . . . . . . . . . . . . . . . . . . . . . . . .93
`Package Control Script Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97
`Package Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . .100
`
`5. Building an HA Cluster Configuration
`
`Preparing Your Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103
`Editing Security Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103
`Ensuring Consistency of Kernel Configuration . . . . . . . . . . . . . . . . .104
`Enabling the Network Time Protocol . . . . . . . . . . . . . . . . . . . . . . . . .104
`
`
`
`5
`
`DHPN-1004 / Page 5 of 260
`
`
`
`Contents
`
`Preparing for Changes in Cluster Size . . . . . . . . . . . . . . . . . . . . . . . 104
`
`Installing MC/ServiceGuard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
`
`Creating the Logical Volume Infrastructure . . . . . . . . . . . . . . . . . . . . 107
`Creating Mirrors of Root Logical Volumes . . . . . . . . . . . . . . . . . . . . 107
`Creating Volume Groups for Mirrored Individual Data Disks. . . . . 109
`Creating Volume Groups on Disk Arrays Using PV Links. . . . . . . . 111
`Deactivating the Volume Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
`Distributing the Volume Group to Other Nodes . . . . . . . . . . . . . . . . 114
`Creating Additional Volume Groups . . . . . . . . . . . . . . . . . . . . . . . . . 116
`
`Configuring the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
`Using SAM to Configure the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . 117
`Using MC/ServiceGuard Commands to Configure the Cluster . . . . 118
`Adding or Removing Nodes While the Cluster is Running. . . . . . . . 121
`
`Verifying the Cluster Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
`
`Distributing the Binary Configuration File . . . . . . . . . . . . . . . . . . . . . 124
`Distributing the Configuration File with SAM . . . . . . . . . . . . . . . . . 124
`Distributing the Configuration File with HP-UX Commands . . . . . 124
`Storing Volume Group and Cluster Lock Configuration Data . . . . . 125
`Checking Cluster Operation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
`
`Preventing Automatic Activation of Volume Groups . . . . . . . . . . . . . . 127
`
`Setting up Autostart Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
`
`Changing the System Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
`
`Deleting the Cluster Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
`
`6. Configuring Packages and Their Services
`
`Creating the Package Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . 132
`Using SAM to Configure a Package . . . . . . . . . . . . . . . . . . . . . . . . . . 132
`Using MC/ServiceGuard Commands to Create a Package . . . . . . . . 133
`
`6
`
`
`
`DHPN-1004 / Page 6 of 260
`
`
`
`Contents
`
`Writing the Package Control Script . . . . . . . . . . . . . . . . . . . . . . . . . . . .138
`Using SAM to Write the Package Control Script . . . . . . . . . . . . . . . .138
`Using Commands to Write the Package Control Script. . . . . . . . . . .138
`Customizing the Package Control Script . . . . . . . . . . . . . . . . . . . . . .139
`Adding Customer Defined Functions to the Package Control Script 142
`Adding or Removing Packages on a Running Cluster . . . . . . . . . . . .143
`
`Verify and Distribute the Configuration . . . . . . . . . . . . . . . . . . . . . . . .144
`Distributing the Configuration File And Control Script with SAM .144
`Copying Package Control Scripts with HP-UX commands . . . . . . . .144
`Distributing the Binary Cluster Configuration File with HP-UX
`Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .144
`Verifying Cluster and Package Configuration . . . . . . . . . . . . . . . . . .145
`
`7. Cluster and Package Maintenance
`
`Managing the Cluster and Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148
`Starting the Cluster When all Nodes are Down . . . . . . . . . . . . . . . . .148
`Adding Previously Configured Nodes to a Running Cluster . . . . . . .149
`Removing Nodes from Operation in a Running Cluster . . . . . . . . . .150
`Halting the Entire Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151
`Reconfiguring a Halted Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .152
`Automatically Restarting the Cluster . . . . . . . . . . . . . . . . . . . . . . . . .153
`
`Reconfiguring a Running Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .154
`Using SAM to Add Nodes to the Configuration
`While the Cluster is Running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155
`Using MC/ServiceGuard Commands to Add Nodes to the Configuration
`While the Cluster is Running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155
`Using SAM to Delete Nodes from the Configuration
`While the Cluster is Running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .156
`Using MC/ServiceGuard Commands to Delete Nodes
`from the Configuration While the Cluster is Running. . . . . . . . . . .156
`
`
`
`7
`
`DHPN-1004 / Page 7 of 260
`
`
`
`Contents
`
`Using SAM to Change the Volume Group Configuration
`While the Cluster is Running. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
`Using MC/ServiceGuard Commands to Change the Volume Group
`Configuration While the Cluster is Running. . . . . . . . . . . . . . . . . . 158
`
`Managing Packages and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
`Starting a Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
`Halting a Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
`Moving a Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
`Reconfiguring on a Halted Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . 161
`
`Reconfiguring a Package on a Running Cluster. . . . . . . . . . . . . . . . . . 162
`Adding a Package to a Running Cluster . . . . . . . . . . . . . . . . . . . . . . 162
`Deleting a Package from a Running Cluster. . . . . . . . . . . . . . . . . . . 163
`Changing the PACKAGE_SWITCHING_ENABLED Flag . . . . . . . . . . . . 163
`Allowable Package States During Reconfiguration . . . . . . . . . . . . . 164
`
`Responding to Cluster Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
`
`Single-Node Operation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
`
`Removing MC/ServiceGuard from a System . . . . . . . . . . . . . . . . . . . . 168
`
`8. Troubleshooting Your Cluster
`
`Testing Cluster Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
`Start the Cluster using SAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
`Testing the Package Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
`Testing the Cluster Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
`Testing the Network Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
`
`Monitoring Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
`Using Event Monitoring Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
`Using HP Predictive Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
`
`Replacing Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
`Replacing a Faulty Array Mechanism . . . . . . . . . . . . . . . . . . . . . . . . 175
`
`8
`
`
`
`DHPN-1004 / Page 8 of 260
`
`
`
`Contents
`
`Replacing a Faulty Mechanism in an HA Enclosure . . . . . . . . . . . . .175
`Replacing a Lock Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .176
`Online Hardware Maintenance with In-line SCSI Terminator. . . . .176
`
`Troubleshooting Approaches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .179
`Reviewing Cluster and Package States. . . . . . . . . . . . . . . . . . . . . . . .179
`Reviewing Package IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . .187
`Reviewing the System Log File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187
`Reviewing Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .188
`Reviewing the Package Control Script . . . . . . . . . . . . . . . . . . . . . . . .188
`Using the cmquerycl and cmcheckconf Commands. . . . . . . . . . . .189
`Using the cmscancl Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . .189
`Using the cmviewconf Command . . . . . . . . . . . . . . . . . . . . . . . . . . .190
`Reviewing the LAN Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . .190
`
`Solving Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192
`Cluster Re-formations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192
`System Administration Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .193
`Package Movement Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .195
`Node and Network Failures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .195
`
`A. MC/ServiceGuard Commands
`
`B. Enterprise Cluster Master Toolkit
`
`C. Designing Highly Available Cluster Applications
`
`Automating Application Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . .208
`Insulate Users from Outages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .208
`Define Application Startup and Shutdown. . . . . . . . . . . . . . . . . . . . .209
`
`Controlling the Speed of Application Failover . . . . . . . . . . . . . . . . . . . .210
`Replicate Non-Data File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . .210
`Use Raw Volumes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .211
`
`
`
`9
`
`DHPN-1004 / Page 9 of 260
`
`
`
`Contents
`
`Evaluate the Use of JFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
`Minimize Data Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
`Use Restartable Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
`Use Checkpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
`Design for Multiple Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
`Design for Replicated Data Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
`
`Designing Applications to Run on Multiple Systems. . . . . . . . . . . . . . 215
`Avoid Node Specific Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
`Avoid Using SPU IDs or MAC Addresses . . . . . . . . . . . . . . . . . . . . . 217
`Assign Unique Names to Applications. . . . . . . . . . . . . . . . . . . . . . . . 217
`Use uname(2) With Care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
`Bind to a Fixed Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
`Bind to Relocatable IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
`Give Each Application its Own Volume Group . . . . . . . . . . . . . . . . . 220
`Use Multiple Destinations for SNA Applications . . . . . . . . . . . . . . . 221
`Avoid File Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
`
`Restoring Client Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
`
`Handling Application Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
`Create Applications to be Failure Tolerant . . . . . . . . . . . . . . . . . . . . 224
`Be Able to Monitor Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
`
`Minimizing Planned Downtime. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
`Reducing Time Needed for Application Upgrades and Patches . . . . 226
`Providing Online Application Reconfiguration . . . . . . . . . . . . . . . . . 227
`Documenting Maintenance Operations . . . . . . . . . . . . . . . . . . . . . . . 228
`
`D. Integrating HA Applications with MC/ServiceGuard
`
`Checklist for Integrating HA Applications . . . . . . . . . . . . . . . . . . . . . . 230
`Defining Baseline Application Behavior on a Single System . . . . . . 230
`Integrating HA Applications in Multiple Systems . . . . . . . . . . . . . . 231
`Testing the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
`
`10
`
`
`
`DHPN-1004 / Page 10 of 260
`
`
`
`Contents
`
`E. Rolling Software Upgrades
`
`Steps for Rolling Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .234
`Performing a Cold Install During Rolling Upgrade . . . . . . . . . . . . . .234
`Using SAM to Perform a Rolling Upgrade . . . . . . . . . . . . . . . . . . . . .235
`Keeping Kernels Consistent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .235
`
`Example of Rolling Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .236
`Step 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .236
`Step 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .237
`Step 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .237
`Step 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .238
`Step 5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .238
`
`Limitations of Rolling Upgrades. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .240
`
`F. Using OTS/9000 on the LAN
`
`G. Blank Planning Worksheets
`
`Blank Hardware Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .244
`
`Blank Power Supply Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .245
`
`Blank Volume Group and Physical Volume Worksheet . . . . . . . . . . . .246
`
`Cluster Configuration Worksheet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .247
`
`Package Configuration Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . .248
`
`
`
`11
`
`DHPN-1004 / Page 11 of 260
`
`
`
`Contents
`
`12
`
`
`
`DHPN-1004 / Page 12 of 260
`
`
`
`Printing History
`
`Printing Date
`
`Part Number
`
`Edition
`
`January 1995
`
`June 1995
`
`December 1995
`
`August 1997
`
`January 1998
`
`B3936-90001
`
`B3936-90003
`
`B3936-90005
`
`B3936-90019
`
`B3936-90024
`
`First
`
`Second
`
`Third
`
`Fourth
`
`Fifth
`
`The last printing date and part number indicate the current edition,
`which applies to the A.11.01 version of MC/ServiceGuard.
`The printing date changes when a new edition is printed. (Minor
`corrections and updates which are incorporated at reprint do not cause
`the date to change.) The part number is revised when extensive technical
`changes are incorporated.
`New editions of this manual will incorporate all material updated since
`the previous edition.
`HP Printing Division:
`
`Enterprise Systems Division
`Hewlett-Packard Co.
`19111 Pruneridge Ave.
`Cupertino, CA 95014
`
`
`
`13
`
`DHPN-1004 / Page 13 of 260
`
`
`
`
`
`14
`14
`
`
`
`DHPN-1004 I Page 14 of 260
`
`DHPN-1004 / Page 14 of 260
`
`
`
`1
`
`MC/ServiceGuard at a Glance
`
`MC/ServiceGuard at a Glance
`
`This chapter introduces MC/ServiceGuard on HP-UX and shows where
`to find different kinds of information in this book. The following topics
`are presented:
`
`• What is MC/ServiceGuard?
`• Using this Guide
`• For More Information
`If you are ready to start setting up MC/ServiceGuard clusters, skip
`ahead to the chapter “Planning and Documenting an HA Cluster.”
`Specific steps for setup are given in the chapter “Building an HA Cluster
`Configuration.”
`
`Chapter 1
`
`15
`
`DHPN-1004 / Page 15 of 260
`
`
`
`MC/ServiceGuard at a Glance
`What is MC/ServiceGuard?
`
`What is MC/ServiceGuard?
`MC/ServiceGuard allows you to create high availability clusters of HP
`9000 Series 800 computers. A high availability computer system
`allows application services to continue in spite of a hardware or software
`failure. Highly available systems protect users from software failures as
`well as from failure of a system processing unit (SPU), disk, or local area
`network (LAN) component. In the event that one component fails, the
`redundant component takes over. MC/ServiceGuard and other high
`availability subsystems coordinate the transfer between components.
`An MC/ServiceGuard cluster is a networked grouping of HP 9000 series
`800 servers (host systems known as nodes) having sufficient
`redundancy of software and hardware that a single point of failure
`will not significantly disrupt service. Application services (individual
`HP-UX processes) are grouped together in packages; in the event of a
`single service, node, network, or other resource failure,
`MC/ServiceGuard can automatically transfer control of the package to
`another node within the cluster, allowing services to remain available
`with minimal interruption.
`Figure 1-1 shows a typical MC/ServiceGuard cluster with two nodes.
`
`Figure 1-1
`
`Typical Cluster Configuration
`
`node 2
`
`pkg B
`
`X (cid:9) 1
`
`node 1
`
`pkg A
`
`pkg A disks
`
`pkg A mirrors
`
`pkg B disks
`
`pkg B mirrors
`
`HUB
`
`16
`
`Chapter 1
`
`DHPN-1004 / Page 16 of 260
`
`
`
`MC/ServiceGuard at a Glance
`What is MC/ServiceGuard?
`
`In the figure, node 1 (one of two SPU’s) is running package A, and node 2
`is running package B. Each package has a separate group of disks
`associated with it, containing data needed by the package’s applications,
`and a mirror copy of the data. Note that both nodes are physically
`connected to both groups of mirrored disks. However, only one node at a
`time may access the data for a given group of disks. In the figure, node 1
`is shown with exclusive access to the top two disks (solid line), and node
`2 is shown as connected without access to the top disks (dotted line).
`Similarly, node 2 is shown with exclusive access to the bottom two disks
`(solid line), and node 1 is shown as connected without access to the
`bottom disks (dotted line).
`Mirror copies of data provide redundancy in case of disk failures. In
`addition, a total of four data buses are shown for the disks that are
`connected to node 1 and node 2. This configuration provides the
`maximum redundancy and also gives optimal I/O performance, since
`each package is using different buses.
`Note that the network hardware is cabled to provide redundant LAN
`interfaces on each node. MC/ServiceGuard uses TCP/IP network services
`for reliable communication among nodes in the cluster, including the
`transmission of heartbeat messages, signals from each functioning
`node which are central to the operation of the cluster. TCP/IP services
`also are used for other types of inter-node communication. (The
`heartbeat is explained in more detail in the chapter “Understanding
`MC/ServiceGuard Software.”)
`
`Failover
`Under normal conditions, a fully operating MC/ServiceGuard cluster
`simply monitors the health of the cluster's components while the
`packages are running on individual nodes. Any host system running in
`the MC/ServiceGuard cluster is called an active node. When you create
`the package, you specify a primary node and one or more adoptive
`nodes. When a node or its network communications fails,
`MC/ServiceGuard can transfer control of the package to the next
`available adoptive node. This situation is shown in Figure 1-2.
`
`Chapter 1
`
`17
`
`DHPN-1004 / Page 17 of 260
`
`
`
`MC/ServiceGuard at a Glance
`What is MC/ServiceGuard?
`
`Figure 1-2
`
`Typical Cluster After Failover
`
`node 1
`
`pkg A disks
`
`pkg A mirrors
`
`pkg B disks
`
`pkg B mirrors
`
`.,
`
`node 2
`
`pkg A
`
`pkg B
`
`C 1-
`
`1
`
`HUB
`
`a a I
`
`After this transfer, the package remains on the adoptive node as long the
`adoptive node continues running, even if the primary node comes back
`online. In situations where the adoptive node continues running
`successfully, you may manually transfer control of the package back to
`the primary node at the appropriate time. In certain circumstances, in
`the event of an adoptive node failure, a package that is running on an
`adoptive node will switch back automatically to its primary node
`(assuming the primary node is running again as a cluster member).
`Figure 1-2 does not show the power connections to the cluster, but these
`are important as well. In order to remove all single points of failure from
`the cluster, you should provide as many separate power circuits as
`needed to prevent a single point of failure of your nodes, disks and disk
`mirrors. Each power circuit should be protected by an uninterruptible
`power source. For more details, refer to the section on “Power Supply
`Planning” in the “Planning” chapter.
`MC/ServiceGuard is designed to work in conjunction with other HP high
`availability products, such as MirrorDisk/UX, which provides disk
`redundancy to eliminate single points of failure in the disk subsystem;
`Event Monitoring Service (EMS), which lets you monitor and detect
`failures that are not directly handled by MC/ServiceGuard; disk arrays,
`which use various RAID levels for data protection; and HP's UPS
`
`18
`
`Chapter 1
`
`DHPN-1004 / Page 18 of 260
`
`
`
`MC/ServiceGuard at a Glance
`What is MC/ServiceGuard?
`
`(uninterruptible power supply), PowerTrust, which eliminates failures
`related to power outage. These products are highly recommended along
`with MC/ServiceGuard to provide the greatest degree of availability.
`
`Chapter 1
`
`19
`
`DHPN-1004 / Page 19 of 260
`
`
`
`MC/ServiceGuard at a Glance
`Using this Guide
`
`Using this Guide
`This manual presents the tasks you need to perform in order to create a
`functioning HA cluster using MC/ServiceGuard. These tasks are shown
`in Figure 1-3.
`
`Figure 1-3
`
`Tasks in Configuring an MC/ServiceGuard Cluster
`
`Plan the Cluster
`
`Set Up Hardware
`
`Configure LVM
`
`- Hardware
`- LVM
`- Cluster
`- Packages
`
`L_> (cid:9)
`
`- SPU
`- Disks
`- LANs
`- Power
`
`L_, (cid:9)
`
`- Lock VG
`- Physical VGs
`- VGs for Packages
`
`L_,>
`
`Chapters 1- 4
`
`Chapter 5
`
`Chapter 5
`
`Configure Cluster
`
`Configure Packages
`
`- Gather Data
`- Edit Cl Config File
`- Check File
`- Send to other nodes
`
`L_, (cid:9)
`
`- Edit Pkg Config File
`- Edit Control Script
`- Check Files
`- Send to other nodes
`
`L_,)
`
`Chapter 5
`
`Chapter 6
`
`Maintenance
`
`- Cluster
`- Packages
`- Hardware
`- Diagnostics
`- Monitors
`
`Chapters 7, 8
`
`The tasks in Figure 1-3 are covered in step-by-step detail in chapters 4
`through 7. It is strongly recommended that you gather all the data that
`is needed for configuration before you start. Refer to Chapter 4, “Planning
`and Documenting an HA Cluster,” for tips on gathering data.
`If you are ready to set up an MC/ServiceGuard cluster, skip ahead to the
`“Planning” chapter. Specific guidelines on cluster creation are found in
`the chapters “Building a Cluster Configuration” and “Configuring
`Packages and their Services.”
`The following is a quick glance at the remaining chapters of this book:
`
`• Chapter 2, “Understanding MC/ServiceGuard Hardware
`Configurations,” describes the hardware configurations used by
`MC/ServiceGuard and provides a general view of how they work
`together.
`• Chapter 3, “Understanding MC/ServiceGuard Software Components,”
`describes the software components of MC/ServiceGuard and shows
`how they function within the HP-UX operating system.
`
`20
`
`Chapter 1
`
`DHPN-1004 / Page 20 of 260
`
`
`
`MC/ServiceGuard at a Glance
`Using this Guide
`
`• Chapter 4, “Planning and Documenting an HA Cluster,” steps
`through the planning process and provides a set of worksheets for
`organizing information about the