A Practical Reference
`for Implementing Fibre
`Channel SANs
`
`IBM-Oracle 1011
`Page 1 of 52
`
`

`
`THE ADDiSON-WESLEY NETWORKING ~ASICS SER~ES
`
`®
`
`Storage Are~
`etworks
`
`IBM-Oracle 1011
`Page 2 of 52
`
`

`
`The Addison-Wesley Networking Basics Series is a set of concise, hands-on guides to
`today’s key technologies and protocols in computer networking. Each book in the
`series covers a focused topic and explains the steps required to implement and work
`with specific technologies and tools in network programming, administration, and
`security. Providing practical, problem-solving information, these books are written by
`practicing professionals who have mastered complex network challenges.
`
`Tom Clark, Designing Storage Area Networks: A Practical Reference for
`Implementing Fibre Channel SANs, 0-201-61584-3
`
`Gary Scott Malkin, RIP: An Intra-Domain Routing Protocol, 0-20~1-43320-6
`
`Geoff Mulligan, Removing the Spam: Email Processing and Filtering,
`0-201-37957-0
`
`Alvaro Retana, Russ White, and Don Slice, EIGRP for IP: Basic Operation
`and Configuration, 0-201-65773-2
`
`Richard Shea, L2TP: Implementation and Operation, 0-201-60448-5
`
`John W. Stewart III, BGP4: Inter-Domain Routing in the Internet,
`0-201-37951-1
`
`Brian Tung, Kerberos: A Network Authentication System,
`0-201-37924-4
`
`Andrew E Ward, Connecting to the Internet: A Practical Guide about
`LAN-Internet Connectivity, 0-201-37956-2
`
`Visit the Series Web site for new title information:
`http://www, awl.comJcseng/networkingbasics/
`
`IBM-Oracle 1011
`Page 3 of 52
`
`

`
`@
`
`A Practical Reference
`for Implementing
`Fibre Channel SANs
`
`Torn Clark
`
`Addison-Wesley
`An Imprint of Addison Wesley Longman, Inc.
`Reading, Massachusetts ¯ Harlow, England ¯ Menlo Park, California
`Berkeley, California ¯ Don Mills, Ontario * Sydney ¯ Bonn
`Amsterdam ¯ Tokyo ¯ Mexico City
`
`IBM-Oracle 1011
`Page 4 of 52
`
`

`
`Many of the designations used by manufacturers and sellers to distinguish their products
`are claimed as trademarks. Where those designations appear in this book, and Addison
`Wesley Longman, Inc. was aware of a trademark claim, the designations have been
`printed in initial capital letters or in all capitals.
`
`The author and publisher have taken care in the preparation of this book, but make no
`expressed or implied warranty of any kind and assume no responsibility for errors or
`omissions. No liability is assumed for incidental or consequential damages in connection
`with or arising out of the use of the information or programs contained herein.
`
`The publisher offers discounts on this book when ordered in quantity for special sales.
`For more information, please contact:
`
`AWL Direct Sales
`Addison Wesley Longman, Inc.
`One Jacob Way
`Reading, Massachusetts 01867
`(781} 944-3700
`
`Visit A-W on the Web: www.awl.com/cseng/
`
`Library of Congress Cataloging-in-Publication Data
`
`Clark, Tom, 1947
`Designing storage area networks : a practical reference for
`implementing Fibre Channel SANs / Tom Clark.
`p. cm. -- (The Addison-Wesley networking basics series)
`Includes bibliographical references.
`1. Computer networks. 2. Information storage and retrieval
`systems. 3. Internetworking (Telecommunication) I..Title.
`II. Series.
`TK5105.5.C547 1999
`004.6--dc21
`
`99-33181
`CIP
`
`Copyright © 1999 by Addison Wesley Longman, Inc.
`
`All rights reserved. No part of this publication may be reproduced, stored in a retrieval
`system, or transmitted, in any form, or by any means, electronic, mechanical, photo-
`copying, recording, or otherwise, without the prior consent of the publisher. Printed in
`the United States of America. Published simultaneously in Canada.
`
`Text printed on recycled and acid-free paper.
`
`ISBN 0201615843
`
`3 4 5 6 7 8 CR 03 02 01 00
`
`3rd Printing March 2000
`
`IBM-Oracle 1011
`Page 5 of 52
`
`

`
`Preface
`
`1.2
`1.3
`
`Text Overview
`Summary
`
`Storage and Networking Concepts
`
`Networking in Front of the Server
`2.1.1
`Serial Transport
`2.1.2 Access Method
`2. t .3 Addressing
`2.1,4 Pachetizing of Data
`2.1.5 Routing of Pachets
`2.1,6 Upper-Layer Protocol Support
`Traditional SCSI Bus Architecture
`Network-Attached Storage
`Networking behind the Server
`Summary
`
`2.2
`2.3
`2.4
`2.5
`
`Chapter 3: Fibre Channel Internals
`
`Fibre Channel Layers
`3.1
`Gigabit Transport
`3.2
`3.3
`Physical-Layer Options
`3.4 Data Encoding
`3.5
`Ordered Sets
`Framing Protocol
`3.6
`3.7
`Class of Service
`3.8
`Flow Control
`3.9
`Name and Addressing Coriventions
`3.10
`Summary
`
`v
`
`ix
`
`1
`4
`6
`
`7
`
`9
`13
`13
`13
`14
`14
`14
`15
`17
`19
`20
`
`23
`
`23
`24
`26
`30
`33
`34
`36
`39
`40
`43
`
`IBM-Oracle 1011
`Page 6 of 52
`
`

`
`4.1
`4.2
`
`4.3
`
`4.4
`4.5
`
`Point-to-Point
`Arbitrated Loop
`Loop Physical Topology
`4.2A
`Loop Addressing
`4.2.2
`Loop Initialization
`4.2.3
`Port Login
`4.2.4
`Loop Port State Machine
`4.2.5
`Arbitration
`4.2.6
`Nonbroadcast Nature of Arbitrated Loop
`4.2.7
`Design Considerations for Arbitrated Loop
`4.2.8
`Fabrics
`Fabric Login
`4.3.1
`Simple Name Server
`4.3.2
`State Change Notification
`4.3.3
`4.3.4
`Private Loop Support
`Fabric Zoning
`4.3.5
`Building Extended SANs
`Summary
`
`Chapter 5:
`
`Fibre Channel Products
`
`5.1
`5.2
`5.3
`5.4
`5.5
`
`5.6
`5.7
`5.8
`5.9
`5.10
`
`Gigabit Interface Converters (GBICs)
`Host Bus Adapters
`Fibre Channel RAID
`Fibre Channel JBODs
`Arbitrated Loop Hubs
`Star Topology for Arbitrated Loop
`5.5.1
`5.5.2 Hub Architecture
`5.5.3 Unmanaged Hubs
`5.5.4 Managed Hubs
`Switching Hubs
`Fabric Switches
`Fibre Channel-to-SCSI Bridges
`SAN Software Products
`Summary
`
`47
`
`47
`49
`50
`52
`55
`62
`63
`64
`66
`68
`75
`79
`8O
`81
`82
`84
`85
`86
`
`89
`
`90
`92
`95
`99
`101
`102
`103
`107
`108
`112
`113
`116
`117
`121
`
`IBM-Oracle 1011
`Page 7 of 52
`
`

`
`Chapter 6: Problem Iso~io~ i~ SANs
`
`6.1
`6.2
`6.3
`
`Simple Problem-Isolation Techniques
`Fibre Channel Analyzers
`Summary
`
`Chapter 7: Management of SANs
`
`7.1
`
`7.2
`7.3
`7.4
`
`7.5
`
`Storage Network Management
`In-Band Management
`7.1.1
`7.1.2 Out-of-Band Management
`7.1.3
`SNMP
`7.l.4 HTTP
`7.1.5 TELNET
`Storage Network Management Issues
`7.1.6
`Storage Resource Management
`Storage Management
`Storage, Systems, and Enterprise Management
`Integration
`Summary
`
`Chapter 8: Application S~udies
`
`8.1
`8.2
`8.3
`8.4
`8.5
`8.6
`8.7
`8.8
`
`Full-Motion Video
`Prepress Operations
`LAN-Free and Server-Free Tape Backup
`Server Clustering
`Internet Service Providers
`Campus Storage Networks
`Disaster Recovery
`Summary
`
`Chapter 9~’Fibre Channel Futures
`
`9.1
`9.2
`9.3
`9.4
`9.5
`9.6
`
`Bandwidth
`Fibre Channel over Wide Area Networking
`Coexistence within Enterprise Networks
`Interoperability
`Total SAN Solutions
`Summary
`
`126
`130
`133
`
`135
`
`137
`138
`139
`140
`143
`144
`144
`146
`147
`
`148
`149
`
`151
`
`151
`154
`156
`161
`163
`165
`167
`168
`
`171
`
`171
`172
`172
`175
`176
`176
`
`IBM-Oracle 1011
`Page 8 of 52
`
`

`
`Appendix A: Fibre Channel Ordered Se~s
`
`Appendix B: Fibre Channel Ver~dors
`
`Bibffography
`
`Glossary
`
`IBM-Oracle 1011
`Page 9 of 52
`
`

`
`Fibre Channel architecture has evolved three distinct physical topolo-
`gies. The first SANs were built on a dedicated, point-to-point connec:
`tion between two devices. Participants in a point-to-point topology
`establish an initial connection via login and then assume full-bandwidth
`availability. Arbitrated Loop allows more than two devices to commu-
`nicate over a shared bandwidth. An initiator in a loop environment
`must negotiate for access to the media before launching a transaction.
`A Fibre Channel fabric topology provides multiple, concurrent, point-
`to-point connections via link-level switching and so entails a much
`higher level of complexity, both in the physical configuration and in
`transport protocol.
`Depending on application requirements, all three topologies may
`be viable for SAN design. Although point-to-point configurations are
`more restrictive, Arbitrated Loop and fabrics offer a wide range of so-
`lutions for a variety of application needs. By incorporating application-
`specific integrated circuits (ASICs), switch and loop hub prices have
`declined while functionality has increased. And by providing attach-
`ment of loop topol.o, gies to fabric ports, it is now possible to design
`and implement complex SANs that efficiently service multiple, some-
`times contending, applications. This was not the case just a few
`years ago.
`
`4.1 Point-to-Point
`
`A point-to-point topology is a simple, direct connection between two
`N_Ports. As shown in Figure 4-1, the transmit leadof one N_Port is
`
`47
`
`IBM-Oracle 1011
`Page 10 of 52
`
`

`
`Figure 4-1
`
`Server
`
`Disks
`A point-to-point lin/e between a server and a disf~
`
`connected via copper or optical cabling to the receive lead of its part-
`ner. The partner’s transmit, in turn, is cabled to the other N Port’s
`receive lead. This cabling scheme creates dedicated bandwidth b~-tween
`the pair, typically 10~0MBps in each direction.
`Before data transactions can occur, the two N_Ports must perform
`an N Port login to assign N_Port addresses, or N_Port IDs. Thereafter,
`a persistent connection is maintained, with utilization of the dedicated
`link determined by the application.
`Although it is conceivable to have an application requiring simul-
`taneous full-duplex transfers, that is, 200MBps total throughput, in
`practice only one side of the link sees any real traffic at a given
`moment. A server and a disk in a point-to-point configuration would
`normally be performing either reads or writes of data but not both
`concurrently. From the server’s standpoint, an ongoing read operation
`would initiate incoming frames on the server’s receiver, with only xc~s
`(for Class 1 or 2 service) leaving the server’s transmitter. Even then, the
`100MBps available bandwidth on the server’s receive link is not likely
`to saturate. Link utilization in point-to-point configurations is deter-
`mined by the performance of the Fibre Channel controllers at either
`end and the buffering available to queue up data to be transmitted or
`received.
`The original point-to-point configurations were based on quarter-
`speed, 266Mbps, bandwidth, with an effective throughput of 25MBps.
`Distributed primarily by Sun Microsystems, quartet-speed imple-
`mentations used fiber-optic GLMs as the transceiver interface to the
`host bus adapter and disk controller logic. Tens of thousands of these
`systems have been shipped to customers over the past several years,
`creating a large but unadvertised base of Fibre Channel products in
`
`IBM-Oracle 1011
`Page 11 of 52
`
`

`
`Chapter 4: SAI~ Topologies
`
`production environments. This legacy base has provided valuable
`experience for improving both the physical transport and protocol
`support.
`As server and disk performance have increased, Fibre Channel
`throughput has superseded quarter and half speed and ~ow provides
`full gigabit bandwidth. At the same time, advances in fabric and Arbi-
`trated Loop technology have enabled more flexibility and functional-
`ity than point to point provides. A point-to-point configuration is still
`viable for simple configurations; for growth of the SAN, however, it is
`important to select the proper HBA and controller components. If the
`vendor includes device drivers or microcode for both point-to-point
`protocol and Arbitrated Loop, accommodating additional devices on
`the SAN can be accomplished with minimal pain.
`
`4.2 Arbitrated Loop
`
`Arbitrated Loop is the most commonly deployed topology for Fibre
`Channel SANs. Loops provide more flexibility and support for more
`devices than does point to point and are more economical per port
`than are fabric switches. Loop-capable HBAs, Fibre Channel disks;
`and Fibre Channel-to-SCSI bridges are also more prevalent than
`fabric-capable devices, primarily because most of these devices have
`already passed through a development and interoperability cycle and
`have emerged as more stable products.
`Arbitrated Loop is a shared, gigabit transport. Like shared Ether-
`net or Token Ring segments, the functional bandwidth available to any
`individual loop device is determined by the total population on the seg-
`ment and the level of activity of the other participants: more active
`talkers, less available bandwidth. An Arbitrated Loop with 50 equally
`active nodes, for example, would provide lOOMBps/50, or only
`2MBps functional bandwidth per node. Arbitrated Loop would there-
`fore’ not be a popular choice for SANs were it not for the fact that a
`typical storage network has relatively few active contenders for band--
`width. Although a single loop may have more than a hundred disk
`drives, there are usually no more than four to six servers initiating
`requests to those drives. Large configurations are thus possible on a
`single loop without dividing the bandwidth down to the level of ordi-
`nary Ethernet.
`
`IBM-Oracle 1011
`Page 12 of 52
`
`

`
`Since the transport is shared, some means must be provided for
`orderly access to the media° In Arbitrated Loop, media access is gained
`through an arbitration protocol. Once an NL_Port has arbitrated and
`won control of the transport, it has the ful! 100MBps bandwidth avail-
`able for its transaction. When the transaction is complete, the NL Port
`closes the temporary connection, making the transport available to
`others.
`
`Arbitrated Loop is a true physical loop, or ring, created by tying the
`transmit lead of one NL_Port to the receive lead of its downstream
`neighbor. The neighbor’s transmit is, in turn, connected to the receiver
`of yet another NL_Port, and so on, until the circle completes at the
`original NL_Port’s receiver. In this way, a continuous data path exists
`through a!l the NL_Ports, allowing any device to access any other
`device on the loop, as illustrated in Figure 4-2.
`The first Arbitrated Loops were built in this fashion, using copper
`or fiber-optic cabling to create the daisy chain of NL Ports. Severa!
`problems quickly arose. Powering o~f or disconnectin~ a single node
`would break the chain and thus crash the loop. A break in cabling or
`faulty transceiver anywhere along the loop would also halt loop traffic
`and entail tedious troubleshooting to locate the problem. Similar to the
`problems encountered in hard~vired Token Ring topologies, the over-
`head and risks associated with dispersed loop cabling promoted the
`development of centralized Arbitrated Loop hubs.
`Arbitrated Loop hubs provide a physical star topology for a loop
`configuration, bringing each NL_Port’s transmit and receive leads to a
`common location. The interna! architecture of a hub completes the
`connections between transmitters and receivers on a port-by-port basis
`via mux (multiplexer) circuitry and finishes the loop by connecting the
`transmitter o/~ the last hub port (for example, port 12) to the receiver
`of the first (for example, port 1}. One of the most useful features of a
`hub is bypass circuitry at each port, which allows the loop to circum-
`vent a disabled or disconnected node while maintaining operation.
`Most unmanaged Arbitrated Loop hubs also validate proper gigabit
`signaling before allowing a device to insert into the loop, whereas
`managed hubs provide additional functionality. These features will be
`described in more detail in Chapter 5.
`
`IBM-Oracle 1011
`Page 13 of 52
`
`

`
`Chapter 4: SAN Topologies 5~
`
`Transmit
`
`Receive
`
`Receive
`
`Transmit
`
`Figure 4-2 A daisy chain Arbitrated Loop
`
`Transmit
`
`Receive
`
`Since an Arbitrated Loop hub supplies a limited number of ports,
`building larger loops may require linking multiple hubs. This is called
`hub cascading. As shown in Figure 4-3, a cascade is simply a normal
`cable connection between a port on one hub and a port on another. No
`special cable is required, although to minimize potential ground loop
`and noise problems, fiber-optic cabling is recommended over copper.
`Cascading consumes one port on the first and last hubs in a chain and
`two ports on intervening hubs. Cascading four 6-port hubs, for exam-
`ple, would yield 18 usable ports, with a fourth of the total ports sacri-
`ficed to achieve the cascade. Depending on the vendor, hubs can be
`cascaded to the Arbitrated Loop maximum of 127 ports, although the
`advisability of doing so should be application-driven. Just because
`some configurations can be built does not mean that they should be.
`Cascading one hub to another extends the loop through the addi-
`tional ports on the downstream hub. A similar effect is achieved by
`inserting a JBOD (just a bunch of disks) into a hub port. Although the
`link between the hub and the JBOD consists of a single cable pair, the
`JBOD itself comprises of a series of Arbitrated Loop disks daisy
`chained (transmit to receive) together. The transmit and receive leads
`of the JBOD interface to the hub represents not a single NL_Port but
`an entire cluster, or loop segment, of multiple NL_Ports. The loop is
`
`IBM-Oracle 1011
`Page 14 of 52
`
`

`
`52
`
`Designing Storage Area Networks
`
`Disks
`
`Hub
`
`I Loop l
`
`Server
`
`Figure 4-3 Cascaded Arbitrated Loop hubs
`
`Loop
`
`IHub
`
`Disks
`
`thus extended through the JBOD enclosure and the loop population
`increased by the number of drives in the JBOD chassis. This is an
`important consideration when calculating hub port requirements and
`optimal loop size.
`Arbitrated Loop standards provide address space for up to 127
`(126 NL_Ports and 1 FL_Port) devices on one loop. Fibre Channel
`specifications allow for 10-km runs over single-mode cabling and long-
`wave fiber-optic transceivers. The reader is advised not to combine
`these two concepts. Even a few ~10-km links on a single loop can
`severely impede loop performance, since each 10-kin link incurs a 50-
`microsecond propagation delay in each direction. Every trdnsaction on
`the loop would have to traverse the extended links, multiplying the
`effect of each transit delay by the number of transactions. Long-haul
`requirements for disaster recovery or campus networks are better
`served with .dedicated switch ports, with attached loops at either end
`for local traffic.
`
`4.2.2 Loop Addressing
`
`An NL_Port, like an N_Port, has a 24-bit port address. If no switch
`connection exists, the upper 2 bytes of this port address are zeroed to
`×’ 00 00 ’. This is referred to as private loop, since devices on the loop
`have no connection to the outside world. If the loop is attached to a
`fabric and an NL_Port supports fabric login, the upper 2 bytes (and
`
`IBM-Oracle 1011
`Page 15 of 52
`
`

`
`Ch~r 4: SAN Tc~po~gi~s
`
`possibly the last byte) are assigned a positive value by the switch. This
`mode is called public loop, since fabric-capable NL_Ports are members
`of both a local loop and a greater fabric community and need a fu11
`24-bit address for identity in the network. In the case of public loop
`assignment, the value of the upper 2 bytes represents the loop identi-
`fier and would be common to all NL_Ports on the same loop that per-
`formed login to the fabric.
`In both public and private Arbitrated Loops, the last byte of the
`24-bit port address is referred to as the Arbitrated Loop Physical
`Address, or AL_PA. The AL PA is acquired during initialization of the
`loop and may, in the case of fabric-capable loop devices, be modified
`by the switch during login. The 1-byte AL PA provides a very compact
`addressing scheme and allo,vs a device’s identity to be included as part
`of a 4-byte ordered set. In fact, an ordered set may include two
`AL_PAs, identifying both source and destination devices. The ordered
`set for Open Full Duplex, for example, is "m2s. 5 DI7.4 AL_PD
`AL_PS," with AL_PD representing the destination address and AL_PS
`representing the source address.
`The total number of AL_PAs available for Arbitrated Loop
`addressing is 127. This number was not determined by rigorous per-
`formance testing on assorted loop topologies; nor was it calculated on
`theoretical throughput given various loop populations. It is based
`instead on the requirements of 8b/10b running disparity between
`frames. As a frame terminates with an end-of-flame character, the EOF
`forces the current r,unning disparity negative. By Fibre Channel stan-
`dard, each transmission word between the end of one frame and the
`beginning of another should also leave the running disparity negative.
`This function is provided by the IDLE ordered set, which has a fixed
`format of K28.5 D21o 4 D2!. 5 D21.5. The special K28.5 leaves run-
`ning disparity positive. The D2 l. 4 leaves the running disparity nega-
`tive. The D21.5 characters used for the last 2 bytes are neutral
`disparity. The net result is a negative running disparity at the end of the
`IDLE transmission word.
`Since the loop-specific ordered sets may incIude AL_PAs in the last
`2 byte positions, negative running disparity is facilitated if these values
`are neutral. In the Open Full Duplex ordered set cited previously, for
`example, the mlv. 4 character following the special K2 8.5 would leave
`the running disparity negative. If the destination and source AL_PAs
`
`IBM-Oracle 1011
`Page 16 of 52
`
`

`
`are neutral disparity, the Open transmission word will leave the run-
`ning disparity negative. This satisfies the requirement for the next start
`of frame (SOF).
`If all 256 possible 8-bit bytes are dispatched to the 8b/10b encoder,
`134 will emerge with neutral disparity characters. Fibre Channel
`claims some of these for special purposes. The remaining 127 neutral
`disparity characters have been assigned as AL_PAs.
`The number "127" is thus not a recommended load for a
`100MBps shared transport. It is simply the maximum number (minus
`reserved values) of neutral disparity addresses that could be assigned
`for loop use. At higher Fibre Channel speeds, such as 400MBps, 127
`active loop participants may be quite reasonable or even considered
`inadequate for some needs.
`Since the AL_PA values are determined on the basis of neutral dis-
`parity, a listing of hex values of AL_PAs seems to jump randomly over
`some byte values and not others. Listed sequentially, the hex value of
`AL_PAs would begin 00, 01, 02, 04, 08, 0F, z0, 17, 18, 1B,
`The gaps in the list represent byte values that, after 8b/10b
`encoding, result in nonneutral disparity characters. This is significant
`for some Fibre Channel disk drives, which allow the user to set
`jumpers or dip switches on a controller card to assign a fixed AL_PA
`manually. Typically, the jumper positions correspond only to an index
`of AL_PA values, not to actual hex values, which is the case with most
`network equipment.
`Arbitrated Loop ass~igns priority to AL_PAs, based on numeric
`value. The lower the numeric value, the higher the priority, AL PA pri-
`orityis used during arbitration to give advantage to initiators, such as
`file servers and fabric loop ports. An FL_Port by default has the
`address x’ 00’, which gives it the highest priority over all other
`NL_Ports. When arbitrating against other devices for access to the
`loop, the FL_Port will always win. This helps ensure that a valuable
`resource, such as a switch, can quickly service the loop and then return
`to fabric duties. During address selection, as shown in Figure 4-4,
`servers typically attempt to take the highest-priority, lowest-value
`AL_PAs, whereas disk arrays take lower-priority, higher-value AL_PAs.
`A server with an AL_PA of x’ 0:L’ will have a statistically higher
`chance of winning arbitration against lower-priority contenders,
`
`IBM-Oracle 1011
`Page 17 of 52
`
`

`
`55
`
`AL_PA x’ 0 2’
`
`~ AL_PA x’ ~,2’
`~ AL_PA x’ E2 ’
`~ AL_PA x’
`
`Disks
`
`Figure 4-4 AL_PA assignment on a small loop
`
`although Arbitrated Loop also provides safeguards against starvation
`of any port.
`An NL_Port’s AL_PA may change with every initialization of the
`loop or reset of the device. On the surface, this may seem disruptive,
`but dynamic address assignment by the topology itself greatly reduces
`administrative overhead. As anyone who has had to reconfigure an IP
`network can testify, offloading low-level address administration to the
`topology is highly desirable. Arbitrated Loop initialization guarantees
`that each attached device will have a unique AL_PA. Potential address-
`ing conflicts are possible only when two separate loops are joined
`together--for example, by cascading two active hubs--without initial-
`ization. Some hub vend~rs have responded to this problem by incor-
`porating an initialization sequence whenever a cascade condition is
`sensed.
`
`4.2.3 Loop
`
`Loop initialization is an essential process for allowing new participants
`onto the loop, assigning AL_PAs, providing notification of topology
`changes, and recovering from loop failure. Following loop initializa-
`tion, the loop enters a stable monitoring mode and begins (or resumes)
`normal activity. Depending on the number of NL_Ports attached to
`the loop, an entire loop initialization sequence may take only a few
`milliseconds. For Sun Solaris servers, a loop initialization may result
`
`IBM-Oracle 1011
`Page 18 of 52
`
`

`
`in a message posted to the event log. For NT servers, it is largely
`ignored. In either case, a loop initialization on an active loop normally
`causes a brief suspension of activity, which resumes once initialization
`is complete.
`A loop initialization may be triggered by a number of causes, the
`most common being the introduction of. a new device. The new device
`could be a former participant that has been powered on or an active
`device that has been moved from one hub port to another.
`A number of ordered sets has been defined to cover the various
`conditions that an NL_Port may sense as it launches the initialization
`process. These ordered sets are Loop Initialization Primitive sequences
`and are referred to collectively as LIPs. An NL_Port issues at least 12
`LIPs to start loop initialization. In the following examples, we will as-
`sume a Fibre Channel host bus adapter installed in a file server.
`
`An HBA that is attached to an active loop and is power cycled
`will, on bootup, start processing the incoming bit stream. The
`presence of valid signal and protocol verifies that the server is
`on an active loop. Because the server was powered down,
`however, the HBA has lost the AL_PA that it was previously
`assigned. That previously assigned AL_PA was stored in a
`temporary register in the HBA, and the register was wiped
`clean by the power cycle. The HBA immediately begins
`transmitting LIP(F7, F7) onto the loop. The x~’V is a reserved,
`neutral disparity character. The first occurrence of x~’7 indicates
`that the HBA recognizes that it is on an active loop. The second
`x~’V indicates that the HBA has no AL_PA.
`
`An HBA that is attached to an active loop is moved from one
`hub port to another. As the cable is unplugged from the hub
`and moved to the other port, the HBA temporarily loses Fibre
`Channel signal. On reinsertion, the HBA sees valid signal
`return and begins processing the bit stream. In this instance, the
`HBA still has its previously assigned AL_PA and so begins
`transmitting LIP(F7, AL_PS) onto the loop. The x~’7 indicates
`that the HBA sees the active loop. The AL_PS is the source
`AL_PA of the LIP, that is, the HBA’s previously assigned
`AL_PA. In this example, the HBA is not issuing LIPs in order to
`
`IBM-Oracle 1011
`Page 19 of 52
`
`

`
`acquire an address but to notify the loop that a topology change
`has occurred.
`
`The receiver of the HBA or the receive cable is broken, and the
`server has been power cycled. In this instance, the HBA does
`not see a valid signal on its receiver and assumes that a loop
`failure has occurred. It also does not recall its previously
`assigned AL_PA. The HBA therefore sfarts streaming L]IP(F8,
`F7) onto the loop. The xF8 is another reserved, neutral
`disparity character that is used to indicate a loop-down state.
`The ×F7 indicates that the HBA has no AL_PA.
`
`In the same scenario, if the HBA still has a previously assigned
`AL_PA, it will issue a LIP{FS, AL_PS). The xF8 indicates that
`the HBA senses loop failure. The AL_PS is the source AL_PA of
`the alert.
`
`Of the conditions listed, the most insidious for Arbitrated Loop
`environments is the LIP(FS) stream. A node issuing a normal LIP(F7)
`will trigger, at most, a temporary suspension of loop operations until
`the initialization process is completed. A node issuing LIP(FS)s, how-
`ever, will continue streaming loop-down alarms as long as it cannot
`recognize loop activity on its receiver. If the node’s transmitter is con-
`nected to an active loop, all NL_Ports will enter a suspended initializa-
`tion state and continue to forward the offender’s LIP(FS) stream, as
`shown in Figure 4-5. Normal loop initialization cannot complete, and
`the loop in fact fails. Thi~ has been another challenge for Arbitrated
`Loop hub vendors. Some have responded with autorecovery policies
`that automatically bypass a port that is streaming LIP(FS)s.
`In addition to loss of signal, an NL_Port may LIP(F8) if no valid
`ordered sets are present on the loop. This may occur if an upstream
`node is corrupting the bit stream, due to excessive jitter or malfunction
`of processing logic. Other conditions may trigger LIPs, including a
`node’s inability to arbitrate successfully for loop access. Arbitrated
`Loop provides a fairness algorithm for media access, but if a partici-
`pant is not playing fairly, others on the loop may LIP to reinitialize a
`level playing field. Arbitrated Loop also provides a selective reset LIP
`that is directed by one NL_Port to another. How the reset is imple-
`mented is vendor-specific, but the selective reset LIP(AL_PD, AL_PS)
`
`IBM-Oracle 1011
`Page 20 of 52
`
`

`
`Designing S~or~e Area NetworRs
`
`, LIP(F8, F7)s
`~
`@
`
`Loop Hub
`
`LIP(F8, F7)s
`
`LIP(FS, F7)s
`
`An NL_Port streaming LIP(F8)s onto the loop
`
`Disk
`
`Disk
`
`may cause the target device to reboot. This allo;vs one NL_Port to
`force a misbehaving NL_Port into a known good state.
`The loop initialization process begins when an NL_Port streams at
`least 12 LIPs onto the loop. As each downstream device receives the
`LIP stream, the device enters a state known as Open-Tn±% which sus-
`pends any current operations and prepares the device for the loop ini-
`tialization procedure. The LIPs are forwarded along the loop until all
`NL_Ports, including the originator, are in an open-InJ_ ~ condition.
`At this point, the NL_Ports need someone to be in charge. Unlike
`Token Ring, Arbitrated Loop has no permanent master to monitor the
`topology. Loop initialization therefore provides a selection process to
`determine which device v~ill be the temporary loop master. Once
`selected, the loop master is responsible for conducting the rest of the
`initialization procedure and returning the loop to normal operation.
`See Figure 4-6.
`The loop master is determined by a subroutine kr~own as the Loop
`Initialization Select Master procedure, or LISM. Each loop device vies
`for the position of temporary master by continuously issuing LISM
`frames, which contain a port type identifier (x’ 00’ for FL_Port, x’ v.F,
`for NL_Port) and its 64-bit World-Wide Name. As a downstream
`device receives a LISM frame from an upstream partner, the device first
`checks the identifier. If the identifier is x’. 00’, a fabric is present, and
`the device ceases is own LISM frame broadcast and begins issuing the
`FL_Port’s LISM frame. If the identifier is a standard NL_Port, the
`downstream device compares the World-Wide Name in the LISM
`
`IBM-Oracle 1011
`Page 21 of 52
`
`

`
`Chapter 4: SAN l"opo~ogies
`
`Normal Loop Activity
`
`Open Initialization
`
`Loop Initialization
`Select Master
`
`AL_PA Assignment
`LIFA/LIPA/LIHA/LISA
`
`Positional Mapping
`LIRP/LILP
`
`Close the Loop
`Enter Monitoring Mode
`
`~ sume Normal Loop Activity
`
`Figure 4-6 Steps in loop initialization sequence
`
`frame to its own. As with AL_PA priorities, the World-Wide Name
`with the lowest numeric value has highest priority. If the received
`World-Wide Name has higher priority, the device ceases its own LISM
`broadcast and begins tran, smitting the received LISM. If the received
`World-Wide Name has lower priority, the device throws the received
`LISM away and continues broadcasting its own LISM. Eventually, a
`node will receive its own LISM frame, indicating that it has the high-
`est priority and is therefore temporary loop master. The node then
`begins transmitting a special ordered set, ARB(F0), to notify the others
`that a gemporary master has been selected; as the ARB(F0) circles back
`to the loop master, the initialization can proceed to the next phase.
`The first task of the temporary loop master is to issue a series of
`four frames tha