N etra ™ ft 1800
`
`White Paper
`
`+Sunmicrosystems
`
`THE NETWORK IS THE COMPUTER'"
`
`Sun Microsystems, Inc.
`901 San Antonio Road
`Palo Alto, CA 94303-4900 USA
`650960-1300
`Fax 650 969-9131
`
`April 1999, Revision 09
`
`

`
`Copyright 1999 Sun Microsystems, Inc., 901 San Antonio Road· Palo Alto, CA 94303 USA. All rights reserved.
`
`This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation, No part of
`this product or document may be reproduced in any form by any means without prior written authorization ofSun and its licensors, if any. Third-party
`software, including font technology, is copyrighted and licensed from Sun suppliers.
`Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other
`countries, exclusively licensed through XIOpen Company, Ltd.
`Sun, Sun Microsystems, the Sun logo, AnswerBook,Java, the Java Coffee Cup, and Solaris are trademarks, registered trademarks, or service marks of Sun
`Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks ofSPARC
`International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.
`The OPEN LOOK and Sun"" Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts
`of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to
`the Xerox Graphical User Interface, which license also covers Sun's licensees who implement OPEN LOOK GUls and otherwise comply with Sun's written license
`agreements.
`RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g) (2) (G/87) and FAR 52.227-19(G/87), or
`DFAR 252.227-7015(b) (G/95) and DFAR 227.7202-3(a).
`DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY
`IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE
`EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
`
`Copyright 1999 Sun Microsystems, Inc.. 901 San Antonio Road· Palo Alto, CA 94303 Etats-Unis. Tous droits reserves.
`Ce produit ou document est protege par un copyright et distribue avec des licences qui en restreignent I'utilisation, la copie, Ia distribution, et la decompilation.
`Aucune partie de ce produit ou document ne peut etre reproduite sous aucune forme, par quelque moyen que ce soit, sans I'autorisation prealable et ecrite de Sun et
`de ses bailleurs de licence, s'il y en a. Le logiciel detenu par des tiers, et qui comprend la technologie relative aux polices de car'acteres, est protege par un copyright et
`licencie par des fournisseurs de Sun.
`Des parties de ce produit pourront etre derivees des systemes Berkeley BSD licencies par I'Universite de Californie. UNIX est une marque deposee aux Etats-Unis et
`dans d'autres pays etlicenciee exclusivement par XIOpen Company, Ltd,
`Sun, Sun Microsystems, Ie logo Sun, AnswerBook, Java, Ie logo Jave Coffee Cup, et Solaris sont des marques de fabrique ou des marques deposees, ou marques de
`service, de Sun Microsystems, Inc. aux Etats-Unis et dans d'autres pays. Toutes Ies marques SPARC sont utilisees sous licence et sont des marques de fabrique ou des
`marques deposees de SPARC International, Inc. aux Etats-Unis et dans d'autres pays. Les produits portant les marques SPARC sont bases sur une architecture
`deveIoppee par Sun Microsystems, Inc,
`L'interface d'utilisation graphique OPEN LOOK et Sun"''' a ete developpee par Sun Microsystems, Inc. pour ses utilisateurs etlicencies. Sun reconnait les efforts de
`pionniers de Xerox pour la recherche et Ie developpement du concept des interfaces d'utilisation visuelle OU graphique pour !'industrie de !'informatique. Sun detient
`une licence non exclusive de Xerox sur !'interface d'utilisation graphique Xerox, cette licence couvrant egalement Ies licencies de Sun qui meltent en place !'interface
`d'utilisation graphique OPEN LOOK et qui en outre se conforment aux licences ecrites de Sun.
`
`CETTE PUBLICATION EST FOURNIE "EN L'ETAT" ET AUCUNE GARANTIE, EXPRESSE OU IMPLICITE, N'EST ACCORDEE, Y COMPRIS DES
`GARANTIES CONCERNANT LA VALEUR MARCHANDE, L'APTITUDE DE LA PUBLICATION A REPONDRE A UNE UTILISATION
`PARTICULIERE, OU LE FAIT QU'ELLE NE SOIl' PAS CONTREFAISANTE DE PRODUIT DE TIERS. CE DENI DE GARANTIE NE S'APPLIQUERAIT
`PAS, DANS LA MESURE OU IL SERAIT TENU JURIDIQUEMENT NUL ET NON AVENU,
`
`THIS PUBLICATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT
`NOT LIMITED TO, ANY EXPRESS OR IMPLIED WARRANTY OF NON-INFRINGEMENT, OR THE IMPLIED WARRANTIES OF
`MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
`
`THIS PUBLICATION COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES MAY BE
`PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES MAY BE INCORPORATED IN NEW EDITIONS OF THIS
`PUBLICATION. SUN MICROSYSTEMS, INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR
`PROGRAM(S) DESCRIBED IN THIS PUBLICATION AT ANY TIME.
`
`CETTE PUBLICATION EST FOURNIE «TELLE QUELLE », SANS GARANTIE D'AUCUNE SORTE, QU'ELLE SOIT EXPRIMEE OU SOUS(cid:173)
`ENTENDUE. Y COMPRIS, MAIS SANS S'Y LIMITER, TOUTE GARANTIE EXPRIMEE OU SOUS-ENTENDUE DE NON-CONTREFAl;:ON OU
`TOUTE GARANTIE SOUS-ENTENDUE QUANT A LA QUALITE LOYALE ET MARCHANDE DU PRODUIT OU A SON APTITUDE A ETRE
`UTILISE A UNE FIN PRECISE.
`CETTE PUBLICATION PEUT CONTENIR DES INEXACTITUDES D'ORDRE TECHNIQUE OU DES FAUTES DE FRAPPE. LES
`INFORMATIONS CONTENUES DANS CETTE PUBLICATION POURRONT ETRE REGULIEREMENT MISES A JOUR; CES
`MODIFICATIONS POURRONT ETRE INTEGREES DANS DE NOUVELLES EDITIONS DE CETTE PUBLICATION. SUN MICROSYSTEMS,
`INC. POURRA APPORTER, A TOUT MOMENT, DES AMELIORATIONS ET/OU DES MODIFICATIONS AU(X) PRODUIT(S) ET/OU AU(X)
`PROGRAMME(S) DECRITS DANS CETTE PUBLICATION.
`
`Please
`Recycle
`
`Adobe PostScript
`
`

`
`Contents
`
`1.
`
`Introduction
`
`1
`
`2. Overview 3
`
`2.1
`
`2.2
`
`2.3
`
`Fault Tolerant Design
`
`3
`
`Solaris on the Netra ft 1800
`
`4
`
`Designed for Installation in the Network
`
`5
`
`3.
`
`System Architecture
`
`7
`
`3.1
`
`3.2
`
`3.3
`
`Physical Features
`
`7
`
`Features of the Netra ft 1800
`
`10
`
`The Core
`
`12
`
`3.3.1
`
`3.3.2
`
`CPUset
`
`12
`
`CPUset Features
`
`16
`
`3.3.3 Motherboard
`
`18
`
`3.3.4 Multi-Processing vs. Fault Tolerance
`
`21
`
`3.4
`
`The I/O Subsystems
`
`22
`
`3.4.1
`
`3.4.2
`
`Drive Chassis
`
`22
`
`Removable Media Modules (RMM)
`
`24
`
`3.5
`
`PCI Interfaces
`
`24
`
`3.5.1
`
`3.5.2
`
`PCI Card Carrier
`
`25
`
`Cable Management
`
`27
`
`iii
`
`

`
`3.6
`3.7
`3.8
`3.9
`
`LAN Connections
`
`27
`
`Console, Alarms and Fans Module (CAF)
`The Power Subsystem 29
`30
`
`Device Driver Hardening
`
`28
`
`32
`
`Operating Environment
`31
`3.9.1
`3.10 Configuration Management System (CMS)
`3.10.1 CMS Operation
`33
`3.10.2 CMS User Interface
`3.10.3 Split Mode
`34
`
`34
`
`4. Telco Features
`
`37
`
`4.1
`4.2
`4.3
`4.4
`4.5
`
`Rack Mounting
`
`Simple Repair
`
`37
`37
`
`Alarms
`
`38
`
`Telco I/O Subsystems
`
`38
`
`NEBS 39
`
`5. Conclusions
`
`41
`
`A. Compliances
`
`43
`
`.;'~i
`
`iv
`
`Netra ft 1800 White Paper' April 1999
`
`

`
`Figures
`
`FIGURE 2-1
`
`Operating Principles 3
`
`FIGURE 3-1
`
`Netra ft 1800 Chassis 8
`
`FIGURE 3-2
`
`Netra ft 1800 Module 9
`
`FIGURE 3-3
`
`Netra ft 1800 CPUset Block Diagram 15
`
`FIGURE 3-4
`
`CPUset Module
`
`16
`
`FICURS!3-5
`
`System Architecture
`
`18
`
`FIGURE 3-6
`
`CPUset Error Checking Process
`
`19
`
`FIGURE 3-7
`
`Faulty DMA Detection
`
`21
`
`FIGURE 3-8
`
`Hard Disk Drive Module (HDD)
`
`22
`
`FIGURE 3-9
`
`PCI Carrier
`
`25
`
`FIGURE 3-10
`
`Cable Management System 27
`
`FIGURE 3-11
`
`Console, Alarms and Fans Module (CAF)
`
`28
`
`FIGURE 3-12
`
`Power Supply Module (PSU)
`
`29
`
`FIGURE 3-13
`
`Software Architecture 30
`
`FIGURE 3-14
`
`Configuration Management System (CMS) Operation 33
`
`FIGURE 3-15
`
`Split Mode 35
`
`v
`
`

`
`vi
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`CHAPTER 1
`
`Introduction
`
`The Netra ft 1800 system is a high-performance, fully fault tolerant server using
`standard Sun technology and operating environment, and particularly d~signed for
`installation in mission-critical areas of the communications and data networking
`infrastructure. It is part of the family of Netra t products from Network Systems that
`are designed to meet the needs of the 'Service Driven Network',
`
`The Netra ft 1800 is fully fault tolerant. The fault tolerant design has eliminated all
`single points of failure; this allows the system to offer uninterrupted service both
`during a fault and during a repair. The system availability is greater than 99,999%,
`which translates to less than five minutes downtime per year.
`
`The system architecture is based on the Sun Ultra Enterprise 450, using one to four
`UltraSPARC™ processors, Because the processing core is electrically identical to the
`Ultra Enterprise 450, the Netra ft 1800 is able to run a standard Solaris kernel. This
`gives the Netra ft 1800 the unique ability to run standard application software
`absolutely unchanged on a fault tolerant hardware platform.
`
`To demonstrate strict compliance with the demanding standards of the
`communications and data networks, the Netra ft 1800 has been fully certified to the
`highest level of the Telcordia Network EqUipment Building System specification
`(NEBS™).
`
`The Netra ft 1800 is the ideal computing platform for installation within the
`communications and data network infrastructure. It combines the ease of use and
`mass-market-tested reliability of a standard commercial computer with the
`particular availability and physical requirements of this environment.
`
`The follOWing sections describe in more detail:
`• The design philosphy of the Netra ft 1800 (Chapter 2 "Overview").
`• The detailed characteristics of the functional hardware and software modules
`which comprise the system (Chapter 3 "System Architecture").
`• A discussion of the particular features proVided for the target markets (Chapter 4
`"Telco Features"),
`
`11
`
`

`
`2
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`CHAPTER 2
`
`Overview
`
`2.1
`
`Fault Tolerant Design
`
`The overriding principle of fault tolerant operation is that the system shall continue
`to proVide service during a fault and during repair. This means there are no single
`points of failure, and that all active modules, including the processors, can be
`removed and replaced online. It means that any hardware fault can be detected and
`masked by switching over to a redundant piece of hardware.
`
`Fault tolerant core
`
`Duplicated I/O domains
`
`Disk
`subsystem
`
`Comms
`and 1/0
`
`Disk
`subsystem
`
`Comms
`and 1/0
`
`FIGURE 2-1 Operating Principles
`
`23
`
`

`
`This is achieved in the Netra ft 1800 by having a processing core that is trusted
`absolutely, and then using that core to check the integrity of the duplicated I/O
`subsystems. This is shown in FIGURE 2-1.
`
`The CPUset in the Fault Tolerant Core contains processor modules (one to four),
`their caches, all the RAM memory for that CPUset, and four PCI buses. The two
`CPUsets are driven by synchronized clocks and are identical to each other in all
`respects. Because their clocks are locked, each transistor on a correctly functioning
`CPUset should be doing the same thing as its partner, on a nanosecond by
`nanosecond basis. If this is not the case, then the Netra ft 1800 detects the difference
`(and therefore error) between CPUsets by comparing the next I/O transaction on the
`PCI buses.
`
`There are four PCI bridges which each take a PCI transaction from each CPUset. If
`they are identical, the transaction is passed onto the physical PCI bus. If it is not then
`an exception handler identifies which CPUset was in error.
`
`If the comparisons were made on each clock cycle, as happens in some Fault
`Tolerant systems from other manufacturers, there would be a noticeable slowing in
`CPU performance as a result. In the Netra ft 1800, the actual comparisons which
`determine if a difference has occurred take place in special pipeline registers on the
`PCI bridge. Because this happens much less often, and is part of the I/O transaction
`itself, there is little impact on system performance. The details of this, and the
`justification for guaranteed I/O, is explained in more detail in Chapter 3 "System
`Architecture" .
`
`Having provided a trusted processor core, we are in a position to use this as a
`platform to run software device drivers which detect and fix failures in the
`duplicated I/O domains. We have, in effect, created a fault free virtual machine
`which always appears perfect to the standard Solaris environment which runs on it.
`
`2.2
`
`Solaris on the Netra ft 1800
`
`One of the most Significant features of the Netra ft 1800 is its ability to run the same
`Solaris kernel as its non fault tolerant counterpart, the Ultra Enterprise E450. The
`Application Programing Interface (API) provided to use programs is identical to the
`standard Solaris interface. This, in turn, means that it can run exactly the same
`application software as the non fault tolerant system.
`
`There are a number of important advantages which this gives to the Netra ft 1800:
`
`4
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`T Mass Market Tested
`
`Solaris, and the applications which run on it, is used in hundreds of thousands of
`sites throughout the world. Any bugs which are reported from this large field
`population are fixed qUickly. A Unix fault tolerant system that uses a specially
`modified kernel may have had hundreds of man years of testing prior to installation,
`but the field population is relatively small. Bugs go unnoticed until it is too late, and
`often the latest versions of applications packages are not readily available.
`
`T Ease of Use
`
`There is no need to write special application software; standard packages run
`without modification.
`
`If software is developed on a regular Solaris desktop system, it will run unchanged
`on the Netra ft 1800 (provided the application does not probe inside the operating
`environment) and can be deployed without additional testing or, worse still, re(cid:173)
`porting to the fault tolerant platform, thus avoiding the potential to introduce errors.
`
`T Ease of upgrade
`
`New versions of Solaris merely need to be tested on the Netra ft 1800, not re-ported
`as would be the case for a specially modified kernel. Indeed, future releases of
`Solaris will start to incorporate many of the techniques developed for the
`Netra ft 1800 as standard.
`'
`
`ObViously there are some special software additions, particularly to the exception
`handler which traps a CPU error, the I/O drivers, and the Configuration
`Management System (CMS) which manages the Netra ft 1800.
`
`More details of the fault tolerant-specific software features of the Netra ft 1800 are
`discussed in Chapter 3 "System Architecture".
`
`2.3
`
`Designed for Installation in the Network
`
`The basic reliability of communications networks is greatly enhanced by the
`demanding standards imposed on equipment which is part of the core
`infrastructure. The standards which particularly apply to computing eqUipment are
`embodied in the Network Equipment Building System (NEBS) devised by Telcordia
`Technologies. The Netra ft 1800 has been not only tested, but fully certified to Level
`3 (the highest level) of the NEBS standard.
`
`The NEBS SR3580 tests cover a number of key areas:
`
`Chapter 2 Overview
`
`5
`
`

`
`T Wide Temperature Range
`
`The Netra ft 1800 will function normally at ambient temperatures between _5° and
`+50°C for 96 hours, and +5°C to +40°C indefinitely.
`
`T Earthquake Zone 4
`
`The system continues to operate while being shaken and vibrated in simulations of
`conditions found in areas of the world such as California and Japan during a serious
`earthquake.
`
`T Electrostatic Discharge (ESD) Immunity
`
`The Netra ft 1800 is protected from electrostatic discharge even during maintenance
`operations. This reduces the possibility of failure, even if normal anti-static
`precautions are ignored.
`
`T Fire Containment
`
`The Netra ft 1800 is built to contain any internal fire, and materials have been
`carefuly selected for minimum flammability to reduce the duration and limit the
`spread of a fire.
`
`The detailed list of tests,v,v'hich comprise NEBS is shown in Section 4.5 "NEBS" on
`page 39, and the list of standards which are covered is shown in AppendiX A
`"Compliances..
`
`6
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`CHAPTER 3
`
`System Architecture
`
`3.1
`
`Physical Features
`
`The Netra ft 1800 is housed in a 438 mm/17 inch width enclosure designed to fit into
`a 19-inch, 23-inch, 24-inch or 600-mm rack mount chassis. It is 1467 mm/58 inches
`high (33U) x 567 mm/22 inches deep and weighs up to 220 kg/484 lb. FIGURE 3-1 on
`.page 3-8 shows the basic construction of the chassis.
`
`'-,.. l7,"
`
`, ..,.,
`
`37
`
`

`
`SIDEA
`
`Removable Media
`- ; : - : - - - - - - - ; - - - - Module A
`(A-RMM)
`
`Power S'upplies
`(A-PSUO to A-PSU2)
`and
`B-PSUO to B-PSU2)
`
`SIDE B
`
`Removable Media
`---7:-------,,,....:::...----- Module B
`(B-RMM)
`
`Hard Disk Drives
`(HDDs) in Disk Drive
`Chassis A (A-DSK)
`
`Console, Alarms
`and Fans module A --~
`(A-CAF)
`
`PCI cards ------'-''---'='''7:
`(A-PCIO to A-PCI?)
`
`CPUset B (B-CPU) --2..!....-
`
`CPUset A (A-CPU)
`
`PCI cards __..!..,!
`(B-PCIO to B-PCI?)
`
`Console, Alarms
`and Fans module B -~~~~'.:'!"
`(B-CAF)
`
`Hard Disk Drives
`(HDDs) in Disk Drive ----=:::;~~~I~[
`Chassis B (B-DSK)
`
`FIGURE 3-1 Netra ft 1800 Chassis
`
`8
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`The processing, storage and I/O elements are all duplicated, and all modules are hot
`swappable.
`
`A module is a discrete element housed in a protective metal enclosure. The module
`enclosure is fitted with ejector/insertion levers which incorporate a power interlock
`switch. Modules are the minimum unit which can be swapped. They are called Field
`Replaceable Units (FRUs). Each module has an EEPROM which identifies it to the
`system Configuration Management System (CMS) and holds service information
`which can be interrogated in the event that a module is returned to a repair center.
`
`Protective metal
`enclosure
`
`Power LED
`Fault LED
`
`EEPROM
`
`Injector/ejector
`--=-=-=======--,-:-------~-- lever with power
`switch interlock
`
`FIGURE 3-2 Netra ft 1800 Module
`
`All modules apart from the motherboards are accessed from the front of the system.
`CPUsets, disk and removable media chassis, PCI modules and power supplies are
`organised into two sides, A and B, and modules associated with each side plug into
`the relevant motherboard. Note that the motherboard is itself a hot swappable
`module.
`
`Chapter 3 System Architecture
`
`9
`
`

`
`3.2
`
`Features of the Netra ft 1800
`
`Availability
`• Fault Tolerant system, no single point of failure, by design.
`• CPUset fail over time within 200 mS.
`•
`I/O failover times are dependent on the I/O subsystem.
`
`CPUsets
`• Dual CPUsets containing processors and memory.
`
`Processors
`• One to four UltraSPARC II modules in each CPUset.
`
`Main memory
`• 256 Mbytes to 4 Gbytes in each CPUset.
`
`Disk
`• Six 3.5 inch drives (lor 1.6 inches high) within each drive chassis.
`• Mirrored with one chassis on each side for fault tolerance.
`
`Ethernet
`• Four internal lOBaseT/100BaseT ports (additional ports can be added via PCI
`cards).
`
`PCl
`• Up to 16 PCI cards, hot-pluggable and dynamically reconfigurable. Uses standard
`PCI cards.
`
`10
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`RAS Features
`• Remote reset and/or power cycle.
`• OS watchdog.
`• Redundant consoles..
`• Redundant modem ports.
`• Module level Fault indicators.
`• Service history EEPROM at module level (except disk drives).
`• Superior error logging.
`• All modules hot replaceable.
`• Three programmable external alarms per side.
`• One dedicated external alarm per side for OS running.
`• One serial port for UPS control per side.
`• Split mode for hot operating system and application upgrade.
`
`Software
`• Solaris 2.6 (or compatible versions) plus hardened drivers.
`• Configuration Management System (CMS) API.
`• SunStorEdge Volume Manager 2.5
`
`Environment
`• NEBS tested:
`• SOT at 1800m altitude.
`• Earthquake zone 4.
`• Flammability by burn test.
`
`Chapter 3 System Architecture
`
`11
`
`

`
`3.3
`
`The Core
`
`3.3.1
`
`CPUset
`
`The CPUset plugs into one of the dedicated CPUset locations in the center of the
`main cabinet. With the other CPUset it forms the self-checking and correcting core of
`the system (see FIGURE 3-3).
`
`3.3.1.1
`
`The UltraSPARC II Processor
`
`The Ultra E450 architecture which is the basis of the Netra ft 1800 processing core is
`a shared memory symmetric multiprocessing system built around the UltraSPARC II
`microprocessor. The UltraSPARC II is Sun's latest generation of the SPARC family
`and the second generation of 64-bit UltraSPARC chips.
`
`The UltraSPARC II processors used in the Netra ft 1800 are individually mounted on
`4 inch x 6 inch module cards, along with associated UPA data buffers and high
`speed SRAM external cache memory. These modules are similar to those used in the
`Netra t 1120 telco server. This modular design facilitates easy system expansion
`(adding additional CPUs), processor upgrades (to future UltraSPARC processors)
`and system service.
`
`The CPUset's main logic board provides for up to four UltraSPARC II CPU modules.
`Each processor slot is supported by a DC-to-DC converter module located on the
`CPUset's logic board. Converters provide the proper core voltage for the CPU chips.
`Separate DC-to-DC converters give the system the flexibility to accommodate future
`generations of UltraSPARC processors.
`
`The Netra ft 1800 system bus, main memory and I/O subsystems have been
`carefully crafted to sustain the high data rates necessary to fully utilize all four
`UltraSPARC II processors, resulting in highly scaleable system performance that is
`remarkably linear from a lightly loaded uniprocessor system to a fully-configured
`four processor system.
`
`3.3.1.2
`
`Ultra Port Architecture System Interconnect
`
`Processor modules communicate with the system controller, main memory and I/O
`subsystem via the system's high-speed five-segment Ultra Port Architecture (UPA)
`crossbar datapath. Two UPA segments support pairs of CPU modules, two support
`system I/O and one proVides the system datapath to main memory. The two CPU
`
`12
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`datapaths are 144 bits wide, with 128 bits for data and 16 bits for error correcting
`code (ECC). The two UPA datapaths that support system I/O are 72 bits wide, with
`64 bits for data and 8 bits for ECC. The buffered memory datapath is 576 bits wide,
`with 512 bits (64 bytes) for data and 64 bits for ECC.
`
`Each UPA segment operates independently of the others and moves data blocks over
`a parallel datapath between client buffers and UPA switch buffers. Transfers are
`completed at a clock rate that is automatically synchronized by the system controller
`to 1/3 or 1/4 of the system's clock rate. With 300 MHz CPUs installed, the UPA
`transfers data at 100 MHz, or one transfer every 10 nanoseconds. CPU segments
`move data from 64-byte crossbar buffers to 8-byte local buffers during eight
`consecutive UPA clock cycles. Once its local buffer is filled with out-bound data, a
`client on a shared UPA segment arbitrates for control of the segment, then transfers
`its data to the segment's switch buffer during the requisite number of clock pulses.
`
`All UPA addressing and arbitration occurs over dedicated, point-to-point signal
`lines which are separate from the datapaths. This arrangement allows arbitration
`and addressing to occur in parallel with data transfers, reducing transfer latency and
`improving datapath utilization.
`
`3.3.1.3
`
`CPUset PCI interfaces
`
`All system communications with storage peripherals and network interface devices
`is mediated by three UPA-to-PCI bridge chips located on the CPUset. Each of these
`bridge chips manages communications between the UPA bus and two industry(cid:173)
`standard Peripheral Component Interconnect (PCI) data buses.
`
`Two of the chips provide the four PCI buses which go to the motherboards. The
`third chip interfaces to the various onboard devices on the CPUset.
`
`3.3.1.4
`
`Four-way Interleaved Memory Subsystem
`
`The Netra ft 1800 supports up to 4 Gbyte of ECC-protected dynamic RAM memory.
`Dual inline memory modules (DIMMs) used by the Netra ft 1800 are the same type
`as those used in the Netra t 112x telco servers and the E450 (144-pin, 60 nS, 5V).
`
`Memory is organized into four banks of four DIMMs. Data moves between memory
`and the UPA crossbar switch buffers over and ECC-protected 64 byte wide datapath.
`The 576-bit memory datapath transfers data to or from all four memory modules in
`a bank in parallel during a read or write operation. A memory read or write
`operation to a single memory location requires approximately 13 UPA clock cycles to
`complete (approximately 130 nS with current memory chips).
`
`All four DIMMs within the same bank must be identical, and can range in capacity
`from 64 Mbytes to 256 Mbytes each. With two banks of identical DIMMs (eight
`memory modules) installed, memory operations are two-way interleaved, reducing
`
`Chapter 3 System Architecture
`
`13
`
`

`
`the average latency for reads and writes almost by half, and nearly doubling the
`memory throughput over non-inteleaved operations. With four banks of identical
`DIMMs installed, memory read-write operations are four-way interleaved, nearly
`doubling throughput again over two-way interleaved operation. With four-way
`interleaving enabled, average memory latency is reduced to approXimately 30 nS,
`yielding an average memory throughput of 1.6 Gbytes/sec. If only one or three
`banks of DIMMs are present, interleaving will not occur.
`
`DIMMs in different banks can have different capacities, although installing memory
`modules of different capacities in different banks will affect memory I/O
`performance. Systems with mixed memory configurations will use all available
`memory addresses, but will operate without memory interleaving. Performance will
`benefit from the addition of memory (higher memory cache hit ratio and reduced
`paging), but the performance improvement will be less than that achieved with
`interleaved operation.
`
`14
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`UltraSPARC II pro-
`cessors
`1 and 2
`
`UPA address bus 0
`(35 bits)
`
`I
`
`....
`
`UPA data bus 0 (128 bits)
`
`UltraSPARC II pro-
`cessors
`3 and 4
`
`r
`
`...
`..
`
`UPA address
`bus 1 (35 bits)
`
`System
`controller
`SC_MP
`
`t
`
`Memory DIMMs (16)
`
`I
`I
`Odd group\
`
`I
`
`\
`
`I
`
`I
`
`Even group
`
`~
`
`Ir
`
`CBT
`switch
`
`..
`...
`
`...
`r
`
`UltraBMX
`x9
`
`Memory data (576 bits)
`
`t
`
`UPA data 3 (64 bits)
`
`UPA data bus 1 (128 bits)
`
`UPA address bus 2 (35 bits)
`
`t
`
`~..
`
`U2P
`bridge 1
`
`UPA data bus 2 (64 bits)
`
`Core 1/0
`ASIC
`
`...
`
`U2P
`bridge 2
`
`,
`
`."'.'=:'.'
`
`~..
`
`U2P
`bridge 3
`
`TOD/NVRAM
`
`Flash PROM
`
`Parallel port
`
`Serial ports
`
`r
`
`FIGURE 3-3 Netra ft 1800 CPUset Block Diagram
`
`A
`
`A
`
`,
`
`,
`Ir
`To motherboards A and B
`Four 32/64-bits 25MHz PCI buses
`
`,
`
`Chapter 3 System Architecture
`
`15
`
`

`
`o
`
`o
`
`3.3.2
`
`CPUset Features
`
`Injector lever
`
`Indicator LEOs --cro
`
`Injector lever
`
`oQ
`
`FIGURE 3-4 CPUset Module
`
`16
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`• One to four processor modules. Each processor module contains one UltraSPARC
`II processor with parity-protected secondary cache and ECC-protected main bus.
`These must be identical on each CPUset within a system.
`• Up to 4 Gbytes of ECC DRAM provided through 16 slots for DIMMs of 256
`Mbytes. The memory can be one, two or four-way interleaved to provide up to 1.6
`Gbytes/sec peak memory bandwidth. The minimum memory increment is 256
`Mbytes (four DIMMs).
`• 2 Mbyte Flash PROM accessed through Ebus2 interface provided by a PCIO ASIC.
`• Maintenance bus slave interface including EEPROM, temperature sensors and fan
`control and status. The temperature of each processor is monitored.
`• 40 Kbyte x 8 battery-backed NVRAM.
`• Two U2P controllers proViding four motherboard PCI buses on two Sides: one
`U2P controller to access the boot PROM.
`• An SC_MP system controller which controls access to and switching on the UPA
`buses.
`• Nine UltraBMX cross-bar switch slices to multiplex the multiple UPA buses.
`• Clock generator circuitry generating phase-accurate multiples of the PCI input
`clock for the UPA, processor and U2P clocks.
`• Three variable speed fans.
`• Front panel indicators LEDs which show:
`• Power - CPUset is powered up.
`• Fault - CPUset has a fault.
`InSync - CPUset is running in lockstep with the other.
`•
`• Split - CPUset is running in split mode.
`• Target - CPUset is in the process of haVing the other CPUset's state copied to it.
`• Diagnostic - (when flashing) operating system is running.
`• Error - CPUset is in EState.
`
`Chapter 3 System ArChitecture
`
`17
`
`

`
`3.3.3
`
`Motherboard
`
`The Netra ft 1800 system has two motherboards which have slightly different
`shapes. The assembly comprises a PCB with the bridge components, UltraSCSl
`controller, PCl bus switching and PCBs for power distribution.
`
`A
`
`B
`
`FIGURE 3-5 System Architecture
`
`Each motherboard module is a complete unit comprising the main circuit board,
`power distribution boards, a series of support bars and a rear cover designed for
`easy replacement and uses captive fasteners. The unit has gUide pins to aid insertion
`and handles to ease the process of fitting. During a motherboard replacement, all the
`components on that side of the system must be released from the front.
`
`A key feature of the motherboards is HotPCl (Sun patent applied for) technology.
`This allows standard PCl cards (in a carrier) to be hot plugged in the Netra ft 1800
`system.
`
`18
`
`Netra ft 1800 White Paper • April 1999
`
`

`
`The motherboards have Power and Fault indicators, which show through the rear
`covers, to aid serviceability. The motherboards include connections for the
`-48V power feed and status indication from the optional AC-DC converter.
`
`The motherboards each have a remote control processor which allows remote reset
`and power control of the side of the system via a serial connection.
`
`3.3.3.1
`
`PCI Bridge
`A fundamental component of the motherboard is the pcr voting bridge which forms
`an interface between the CPUsets and the I/O devices. Each I/O transaction is
`monitored here to detect discrepancies in CPU cycles which would indicate a CPU
`failure, and address discrepancies which would indicate an I/O failure. Either of
`these conditions trigger a fault recovery process.
`
`CPUset Error Checking
`
`As noted earlier, the CPUsets have their clocks synchronised by a phase locked loop.
`They effectively run in lockstep. However, it is important to note that although the
`CPUsets are operating in lockstep, they are not hampered by comparing each other
`for errors at each clock cycle. The comparisons are made as part of an I/O
`transaction,',y"hich is a much less frequent occurrence than the processor clock cycle,
`and because'the comparison is part of the I/O transaction there is no impact on
`processor performance.
`
`The sequence of events which are part of this process is shown FIGURE 3-6.
`
`PCI BRIDGE
`
`FREEZE OR
`ALLOW TRANSACTION?
`
`HOT PCI
`(1 of 4)
`
`CPUsetA
`PCI
`
`EXCEPTION
`HANDLER
`
`PCI
`
`CPUset B
`
`FIGURE 3-6 CPUset Error Checking Process
`
`Chapter 3 System Architecture
`
`19
`
`

`
`Each CPUset has four PCI bus interfaces which are cross linked to four PCI voting
`bridges (only one is shown in the diagram with its connections to two CPUsets).
`Each time there is an I/O transaction, the two CPUsets submit what they want to be
`put on to the physical PCI bus which emerges from each bridge. If the two
`transactions are the same, the information is committed to the PCI bus. If they are
`not the same, the transaction is frozen in the pipeline registers and the CPUsets both
`(independently) run the exception handler.
`
`The exception handler starts a very rapid, but comprehensive self-checking routine.
`The CPUset hardware has built-in diagnostics which can indicate faults such as
`parity errors and memory ECC errors, and software diagnostic routines check
`various other functions. The system maintains a CPUset history log which records
`unexpected events such as user core dumps; these count against the CPUset.
`
`All the tests produce a numerical result. The larger the number, the more serious the
`effect of that test failure. The results are pooled and effectively give each CPUset a
`fitness score.
`
`A dead CPUset scores an infinitely large number and is obvious. A failing CPUset
`may still respond but is identified by its higher score. Once the tests are completed,
`the winning CPUset gains control and the I/O transaction from the good CPUset
`completes. This whole process completes in less than 200 mS. The other CPUset is
`marked as bad and can be removed and replaced with a spare module.
`
`GPU?etfailure is therefore transparent to the I/O device and the driver.
`
`Processor Re-Integration (PRI)
`
`When a faulty CPUset is re