(19) United States
`(12) Patent Application Publication
`Fung
`
`(10) Pub. No.: US 2002/0004912 Al
`Jan. 10, 2002
`(43) Pub. Date:
`
`1111111111111111 IIIIII IIIII 11111 1111111111 11111 11111 111111111111111 IIIII IIII
`US 20020004912Al
`
`(54) SYSTEM, ARCHITECTURE, AND METHOD
`FOR LOGICAL SERVER AND OTHER
`NETWORK DEVICES IN A DYNAMICALLY
`CONFIGURABLE MULTI-SERVER
`NETWORK ENVIRONMENT
`
`(75)
`
`Inventor: Henry T. Fung, San Jose, CA (US)
`
`Correspondence Address:
`R. Michael Ananian
`FLEHR HOHBACH TEST
`ALBRITTON & HERBERT LLP
`Four Embarcadero Center, Suite 3400
`San Francisco, CA 94111 (US)
`
`(73)
`
`Assignee: Amphus, Inc.
`
`(21)
`
`Appl. No.:
`
`09/860,212
`
`(22)
`
`Filed:
`
`May 18, 2001
`
`(63)
`
`Related U.S. Application Data
`
`Continuation of application No. 09/121,352, filed on
`Jul. 23, 1998, now Pat. No. 6,079,025. Continuation
`of application No. 08/767,821, filed on Dec. 17, 1996,
`now Pat. No. 5,892,959. Continuation of application
`No. 08/460,191, filed on Jun. 2, 1995, now aban(cid:173)
`doned. Continuation of application No. 08/285,169,
`filed on Aug. 3, 1994, now abandoned. Continuation
`of application No. 08/017,975, filed on Feb. 12, 1993,
`now Pat. No. 5,396,635. Continuation of application
`No. 07/908,533, filed on Jun. 29, 1992, now aban(cid:173)
`doned. Continuation of application No. 07/532,314,
`filed on Jun. 1, 1990, now abandoned. Continuation
`
`of application No. 09/558,473, filed on Apr. 25, 2000.
`Non-provisional of provisional application No.
`60/283,375, filed on Apr. 11, 2001. Non-provisional
`of provisional application No. 60/236,043, filed on
`Sep. 27, 2000. Non-provisional of provisional appli(cid:173)
`cation No. 60/236,062, filed on Sep. 27, 2000.
`
`Publication Classification
`
`Int. Cl.7 ......... G06F 1/26; G06F 1/28; G06F 1/30
`(51)
`(52) U.S. Cl. .............................................................. 713/300
`
`(57)
`
`ABSTRACT
`
`Network architecture, computer system and/or server, cir(cid:173)
`cuit, device, apparatus, method, and computer program and
`control mechanism for managing power consumption and
`workload in computer system and data and information
`servers. Further provides power and energy consumption
`and workload management and control systems and archi(cid:173)
`tectures for high-density and modular multi-server computer
`systems that maintain performance while conserving energy
`and method for power management and workload manage(cid:173)
`ment. Dynamic server power management and optional
`dynamic workload management for multi-server environ(cid:173)
`ments is provided by aspects of the invention. Modular
`network devices and integrated server system, including
`modular servers, management units, switches and switching
`fabrics, modular power supplies and modular fans and a
`special backplane architecture are provided as well as
`dynamically reconfigurable multi-purpose modules and
`servers. Backplane architecture, structure, and method that
`has no active components and separate power supply lines
`and protection to provide high reliability in server environ(cid:173)
`ment.
`
`68
`
`69
`ELECTRIC ~
`50
`SERVICE ~ \
`I**
`/
`I
`\
`I**
`FACILITIES COOLING
`
`ROUTER
`
`52
`
`52
`
`EXTENDTO •••
`MULTIPLE RACKS W BAYS 53
`
`65
`
`64
`
`NAS
`
`ELECTRIC SERVICE
`
`54
`
`-q==L-----~
`
`66
`
`PayPal Ex. 1013, p. 1
`PayPal v. IOENGINE
`
`

`

`'"""' ~=
`~ ?
`.... 0 =
`~ ....
`O' -....
`~
`.... 0 =
`~ ....
`~ "Cl -....
`~ .... ~ = ....
`
`I")
`
`I")
`
`""C
`
`'"""'
`>
`'"""' N
`\0
`,i;;..
`0
`0
`0
`~
`0
`0
`N
`'JJ.
`d
`
`'"""'
`\0
`'"""' 0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`
`NAS
`
`64
`
`(LOCALAND REMOTE)
`MANAGEMENT NODE(S)
`
`65
`
`D
`
`---
`
`54
`
`53
`
`54
`
`62 -
`
`55
`60
`61
`O O * 11 I --
`
`l:Z::II
`
`UPLINK
`
`LOAD BALANCER
`
`ROUTER
`
`I * *
`1I **
`SERVICE ~ 50\
`ELECTRIC ~
`
`==l
`ELECTRIC SERVICE
`
`__ ..
`
`66
`'
`
`MULTIPLE RACKS W BAYS 53
`EXTENDTO •••
`
`52
`
`52
`
`FIG.1
`
`PayPal Ex. 1013, p. 2
`PayPal v. IOENGINE
`
`FACILITIES COOLING
`
`\
`
`69
`
`I
`
`_
`
`/
`
`68
`
`

`

`'"""' ~=
`~ ?
`.... 0 =
`~ ....
`O' -....
`~
`.... 0 =
`~ ....
`~ "Cl -....
`~ .... ~ = ....
`
`I")
`
`I")
`
`""C
`
`'"""'
`>
`'"""' N
`\0
`,i;;..
`0
`0
`0
`~
`0
`0
`N
`'JJ.
`d
`
`'"""'
`\0
`0 ....,
`N
`~ ....
`'JJ. =(cid:173)~
`
`FIG. 2
`
`GROUP
`SERVER
`
`PayPal Ex. 1013, p. 3
`PayPal v. IOENGINE
`
`N
`0
`0
`N
`
`RAM
`
`CPU
`
`BACKPLANE
`
`55
`
`2
`#2
`F1N
`BANK
`FAN
`FAN
`O
`57 ~AN
`
`ISWITCH
`
`MMI
`
`59
`
`SUPPLY
`POWER
`56
`
`#2
`
`SUPPLY
`POWER
`56
`
`#1
`
`ELECTRICAL POWER AND
`
`DATA CONNECTORS
`
`IMM
`
`!SWITCHI
`
`59
`
`2
`#1
`F1N
`FAN
`BANK
`FAN
`O
`N 57
`
`~
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 3 of 19
`
`US 2002/0004912 Al
`
`0
`
`0
`
`0
`
`0::
`r:oLU
`LU~
`S:LU
`c.n
`
`0
`
`0
`
`0
`
`0::
`c:oLU
`LU~
`:S:LU
`c.n
`
`LUO::
`:cLU
`c....,~-
`
`c'.3~
`
`LUO::
`
`:cLLI Ofi::ER-
`(3LU c.n
`
`__. 0::::
`-LU
`<C~
`::a,LU
`LUcn
`
`0
`
`0
`
`0
`
`-,
`
`I
`I
`I
`I
`
`,_ -·
`
`~
`~
`
`i=;
`~
`
`C":)
`
`a...
`a...
`<C
`
`C"-1
`a...
`CL
`<C
`
`a...
`~
`
`-
`
`LUw
`c:>:;z'
`~!:!2
`C>Cl
`tr.;¢
`!:!:!..
`
`(\')
`CJ
`LL
`
`PayPal Ex. 1013, p. 4
`PayPal v. IOENGINE
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 4 of 19
`
`US 2002/0004912 Al
`
`c:::~
`l.Uo:::
`1-----1 8 C, 1---.---1
`:z::z: s
`
`~b::
`~~
`~b::
`~~ =
`ol:i::
`- c ,
`enc...
`
`c=
`C>
`~
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`~~
`L.Uo::: 80
`==
`s
`
`~b::
`~~
`
`z
`
`C,
`~
`
`(cid:127)
`
`-.::::1"'
`-
`CJ
`LL
`
`~ ....-en
`1--_;.,--l 8 C, 1------'l--l
`==
`•--
`s
`
`LUO:::
`
`I
`I
`I
`.J
`
`PayPal Ex. 1013, p. 5
`PayPal v. IOENGINE
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 5 of 19
`
`US 2002/0004912 Al
`
`I-Cl~
`== (cid:173)
`L.1.Jo=
`:a;:;("JL.1.J
`ClL.1.J:aE
`Oenu.J
`COu.........1
`i:fjo~
`I
`--- ------ ------ --------_, ___________ ------------------ ---.
`' I
`c::: =-~
`~oc::
`::a;, ,-
`' ' --~----------------------------
`
`--,
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I--
`
`LLien=
`::a;,0:::LU
`ClU::~
`Ou...LU
`
`LU
`
`1
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`0
`
`0
`
`c:::
`coLU
`L.1.J~
`:S:LU
`en
`
`0
`
`0
`
`L.1.J c:::
`=L.1.J
`<J~--
`~~
`
`0
`
`0
`
`c:::
`caUJ
`LLI5::
`s=:L.1.J en
`
`0
`
`0
`
`0
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`----------------------------------'
`-------------------------------------------------------
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`LO .
`(!)
`LL
`
`PayPal Ex. 1013, p. 6
`PayPal v. IOENGINE
`
`

`

`'"""'
`>
`'"""' N
`\0
`,i;;..
`0
`0
`0
`~
`0
`0
`N
`rF.J.
`d
`
`'"""'
`\0
`0 ....,
`~ ....
`rF.J. =(cid:173)~
`
`O'I
`
`UPLINK
`GIGABIT
`
`'"""' ~=
`~ ?
`.... 0 =
`~ ....
`O' -....
`~
`.... 0 =
`~ ....
`t "Cl -....
`~ .... ~ = ....
`
`I")
`
`I")
`
`""C
`
`N
`0
`0
`N
`
`120-1(102A)
`
`(WITH REDUNDANCY)
`GIGABIT UPLINK
`
`124
`
`(WITH REDUNDANCY)
`GIGABIT UPLINK
`
`PRIMARY SWITCHING FABRIC (SWITCH MODULE)
`
`----. --
`r------------....... -126
`140
`
`NODE QJ i
`
`146
`
`POTS
`\..
`148
`
`142-,,:
`
`I
`I
`I
`I
`
`II
`
`IHJ
`
`l I
`
`/ 11,r---~-:-1--1;,:;~~=-~~~=:-:-!l:-:----1=-...L_J..__l_.:.l..!.i..:i;
`, --
`
`112b-16
`
`SECONDARY SWITCHING FABRIC (SWITCH MODULE)
`
`112b-1
`
`i 1
`
`---
`L
`
`11
`
`-~,
`
`, ,
`
`REMOTE DIAL-IN
`
`144
`MANAGEMENT NODE :
`
`104b-2
`
`Fl G. 6 108b-1108b-2
`
`PayPal Ex. 1013, p. 7
`PayPal v. IOENGINE
`
`100 ;
`
`n~-,-1'~,-.;..;.;J)-, ~ §~~NCER
`
`128
`ROUTER
`130
`
`-132
`
`MANiicEiENT 138
`
`MANAGEMENT NODE
`REMOTE INTERNET
`
`134
`
`INTERNET CONNECTION
`
`136
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 7 of 19
`
`US 2002/0004912 Al
`
`j
`
`~
`~
`
`- -
`
`...
`
`I(cid:173)
`LU
`~ 0:: :z
`~ LU C:::
`>LU
`:::c
`0
`I-
`LU
`
`0
`en
`
`-
`
`-
`
`~ 0
`<(
`0::::
`0
`c5
`LU
`.....I
`< 0 m Cl)
`:::::,
`c::: UJ >-
`LU 0
`LU 0
`en >
`::::c::::: ~
`
`I i l l
`
`I'-
`
`(cid:127)
`
`(9 -LL
`
`-~
`
`
`
`~ -
`-
`
`Cl)
`
`0
`
`0
`
`0
`
`~ -
`en
`
`~ -
`-
`
`Cl)
`
`-
`
`-
`
`1
`
`Cl)
`:::::,
`m
`c..:> a...
`~
`
`PayPal Ex. 1013, p. 8
`PayPal v. IOENGINE
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 8 of 19
`
`US 2002/0004912 Al
`
`INTERNET
`(OR OTHER
`NEnrl/OR~
`
`ROUTER
`
`GIGABIT
`(GS1) SWITCH
`
`0
`
`0
`
`0
`
`0
`
`GIGABIT
`SWITCH
`(652)
`
`FIG. 8
`
`PayPal Ex. 1013, p. 9
`PayPal v. IOENGINE
`
`

`

`'"""' ~=
`~ ?
`.... 0 =
`O' -....
`~
`.... 0 =
`~ "Cl -....
`~ = .....
`~ .....
`""C
`
`~ .....
`
`I")
`
`~ .....
`
`I")
`
`'"""'
`>
`'"""' N
`\0
`,i;;..
`0
`0
`0
`~
`0
`0
`N
`'JJ.
`d
`
`'"""'
`\0
`0 ....,
`\0
`~ .....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`
`402-N
`
`402-1 I ,401
`
`FIG. 9
`
`CPU
`
`NIC
`
`ETHERNET
`
`442-N
`
`PayPal Ex. 1013, p. 10
`PayPal v. IOENGINE
`
`I µC HSM CONTROLALGORITHMI
`MANAGEMENT MODULE 1 (OR DELIVERED UNITS) I ,430
`
`I "'+ 432
`
`AND UNIT
`
`CPU CLOCK FREQ. GENERATOR/CONTROL UNIT N
`
`CPU VOLTAGE GENERATOR/CONTROL UNIT N
`
`CODE
`CPU
`CODE
`CPU
`
`422-1
`CODE CPU CLOCK FREQUENCY GENERATOR/CONTROL UNIT 2
`CPU
`CODE
`
`420-1
`
`CPU VOLTAGE GENERATOR/CONTROL UNIT 1
`
`416-1
`
`404-1 CPU 418-1
`440
`
`I mv r
`
`I
`
`406-N
`
`412-N
`GENERATOR
`I (cid:141) I I INDICATOR
`ACTIVITY
`CPUN
`
`I
`
`n111
`
`I
`
`406-1
`
`GENERATOR
`INDICATOR
`ACTIVITY
`CPU 1
`
`412-1
`7 (cid:141)
`
`SUSPEND/RESUME
`410_N
`
`[
`
`INDICATOR($)
`ACTIVITY
`
`i-,r408-N
`
`MEMORY
`
`SERVER MODULE N
`
`-. SUSPEND/RESUME
`410_1
`
`I/ ~~fo~s1 [
`
`i-,r408-1
`
`MEMORY
`
`SERVER MODULE 1
`
`442-1 :
`
`~-
`
`(OR .. : ... BUS)
`AMPCBUS I
`
`

`

`'""" ~=
`~ ?
`.... 0 =
`~ ....
`O' -....
`~
`.... 0 =
`~ ....
`~ "Cl -....
`~ .... ~ = ....
`
`I")
`
`I")
`
`""C
`
`'"""
`>
`'""" N
`\0
`,i;;..
`0
`0
`0
`~
`0
`0
`N
`'JJ.
`d
`
`'"""
`\0
`0 ....,
`'""" 0
`~ ....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`
`! ~301
`'
`
`----------------L--· 302-1
`
`324
`
`322
`
`vwvruvf\lL
`
`326 ~.............,......., CORE LOGIC CLK _.+ 352
`
`CPU CORE VOLTAGE
`
`(Vcc'CPU)
`
`I
`
`I
`
`[SM 302]
`
`. SIGNAL 338
`
`:
`i
`
`CPU clocK
`
`:
`!
`:
`I 340
`:
`I CLOCK v342
`!
`. GENERATOR I
`:
`350 .-'·'-----'·'-,
`:
`I
`I
`.. I
`\
`REGULATOR VIDEO CLK-356 :
`NIC CLK -354 :
`VOLTAGE
`
`------------------------
`:
`µC
`:
`
`SUSPEND/RESUME
`
`MICROCONTROLLED
`
`RESUME
`
`,378
`xsus.
`
`f ,..-376
`I
`374
`
`._______. 314 ._____.
`
`BIOS
`f ,,-378
`
`FIG. 10
`
`MANAGEMENT
`
`MODULE
`
`AMPC
`302-2
`------------------------
`
`302-N
`
`------
`
`308
`
`306
`
`VIDEO CLK
`
`l---356
`
`SUSPEND
`
`370--1
`
`366
`310 J:1368
`
`NIC CLK
`
`RESUME
`
`PayPal Ex. 1013, p. 11
`PayPal v. IOENGINE
`
`SWITCH
`
`332
`
`I
`
`~362
`
`360
`
`PCIBUS
`
`304
`
`OR µC
`
`....,_.,.....____,,.......,
`
`PMU ----FROM RTC OR NIC
`SUSPEND/RESUME
`
`---------•
`
`I
`
`CORE LOGIC
`
`' I
`' ' I
`' I
`--------------------------------------------------------
`
`330
`
`.
`
`I.
`
`336
`~--~-~-
`I
`
`I~
`
`' . --·· .,_. '
`
`339
`
`329
`
`320
`
`I
`328
`
`RAM
`
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`
`

`

`'"""' ~=
`~ ?
`.... 0 =
`~ ....
`O' -....
`~
`.... 0 =
`~ ....
`~ "Cl -....
`~ .... ~ = ....
`
`I")
`
`I")
`
`""C
`
`'"""'
`>
`'"""' N
`\0
`,i;;..
`0
`0
`0
`~
`0
`0
`N
`'JJ.
`d
`
`'"""'
`\0
`'"""' 0 ....,
`'"""'
`~ ....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`
`FIG. 11
`
`ETHERNEL--265
`
`BUS
`
`257
`
`229
`
`PayPal Ex. 1013, p. 12
`PayPal v. IOENGINE
`
`VIDEO SUB-SYSTEM
`
`236
`
`I
`
`v
`
`: 1
`7
`
`j
`
`\
`
`I FAST ETHERNET I WOL 2311 l I L7u '<:v a 'f I ''. '
`
`'
`
`CONTROLLERS
`
`233
`
`)I x16 DOR SDRAM
`
`I K:
`
`--------------------
`
`208
`
`,-'
`102
`
`207
`SM-BUS
`
`222
`222
`
`x16 SOR SDRAM
`
`IMFORMATION
`MANAGEMENT
`
`BASE
`
`r------1..-
`
`MIB
`
`221
`
`

`

`'"""' ~=
`~ ?
`~ .... ... 0 =
`O' -...
`~
`~ .... ... 0 =
`~ "Cl -...
`~ .... ~ = ....
`
`'"""'
`>
`'"""' N
`\0
`,i;;..
`0
`0
`0
`~
`0
`0
`N
`'JJ.
`d
`
`'"""'
`\0
`0 ...,
`'"""' N
`~ ....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`
`I")
`
`I")
`
`""C
`
`MM~
`: (256)
`
`SERVER MODULE Lr 112a-16
`SERVER MODULE f/ 112a-15
`SERVER MODULE f/ 112a-14
`SERVERMODULE f/112a-13
`SERVER MODULE f/ 112a-12
`SERVER MODULE f/ 112a-11
`SERVERMODULE f/112a-10
`SERVER MODULE f/ 112a-9
`SERVER MODULE I-' 112a-8
`SERVER MODULE f/ 112a-7
`SERVER MODULE f/ 112a-6
`SERVER MODULE V 112a-5
`SERVER MODULE I-' 112a-4
`SERVER MODULE f/ 112a-3
`SERVER MODULE f/ 112a-2
`SERVER MODULE r 112a-1
`
`FIG.12
`
`'ALL LINES REPRESENT DIFFERENTIAL PAIR SIGNALS'
`
`'I
`
`MANAGEMENT
`
`108a-2
`MODULE
`
`MANAGEMENT
`SWITCH/CPU
`
`POWER, RESET, SERIAL PO,,RT, & WATCH DOG
`
`PayPal Ex. 1013, p. 13
`PayPal v. IOENGINE
`
`

`

`'"""' ~=
`~ ?
`.... 0 =
`~ ....
`O' -....
`~
`.... 0 =
`~ ....
`~ "Cl -....
`~ .... ~ = ....
`
`I")
`
`I")
`
`""C
`
`'"""'
`>
`'"""' N
`\0
`,i;;..
`0
`0
`0
`~
`0
`0
`N
`'JJ.
`d
`
`'"""'
`\0
`0 ....,
`'"""' ~
`~ ....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`
`520-1
`
`518-1
`
`516-1
`
`/524
`
`R/L
`
`FIG. 13
`
`L------------J
`
`~~
`
`502
`
`501
`
`510-2
`
`510-1
`
`PayPal Ex. 1013, p. 14
`PayPal v. IOENGINE
`
`,
`i
`-----,,
`
`/
`
`I
`I
`
`INTERNETWORK
`
`---------------------------------------
`'
`'
`r
`/_,,
`
`'
`\
`
`)
`530
`
`_______________________________ J
`
`512-2
`
`512-1
`
`514-1
`
`------------.......
`
`,, .... -.. ----------
`
`R/L
`
`~--------------L----------------1
`
`I
`
`522
`
`I
`I
`I
`
`R/L
`
`____ = OUT OF BAND (AMPC)
`_ =IN BAND
`Mz= TYPE 2 MASTER
`M1 = TYPE 1 MASTER
`$=STREAMING MEDIA SERVER
`$ = CACHE SERVER
`W = WEB SERVER
`R/L = ROUTER/LOAD BALANCER
`LEGENDS:
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 14 of 19
`
`US 2002/0004912 Al
`
`I
`I
`
`' ' I
`I ' I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`------(cid:157)------
`
`' I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`--------------------------------+----------------~------
`'
`'
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`C)
`LL
`
`PayPal Ex. 1013, p. 15
`PayPal v. IOENGINE
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 15 of 19 US 2002/0004912 Al
`
`FIG. 15
`
`PayPal Ex. 1013, p. 16
`PayPal v. IOENGINE
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 16 of 19 US 2002/0004912 Al
`
`I
`
`I
`
`.....
`
`\
`..
`
`,
`
`..,
`
`1 :--,
`'
`'
`..................
`L
`"\'.,\
`' '
`A'
`'\ \
`\ ' '' I \
`I '' \
`\:
`\.\ : :
`..
`',, T:\
`'•·0
`
`•'
`
`I
`I
`
`I
`I
`
`,
`\
`
`FIG. 16
`
`FIG.17
`
`FIG.18
`
`FIG. 19
`
`FIG. 20
`
`FIG. 21
`
`FIG. 22
`
`SUPERVISORIMAPTERIMOVE
`CONTROL SIGNALS
`
`I,,
`•
`~ 0
`Ci) CD Ci)
`
`. ..
`
`...
`
`• 0
`~ ~CD
`
`FIG. 23
`
`SERVER MOD 1
`
`SERVER MOD 2
`SERVER MOD 3
`FIG. 24
`
`' 0
`~
`SERVER MOON
`
`Ci)
`
`PayPal Ex. 1013, p. 17
`PayPal v. IOENGINE
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 17 of 19 US 2002/0004912 Al
`
`HOST COMPUTER
`HOST
`PROCESSOR
`
`102
`....
`
`100
`101 I
`V
`
`103 -
`
`(109
`
`MEMORY
`(RAM)
`
`BUSS
`
`v108
`
`RAID
`CONTROLLER
`I PROCESSOR I
`L.-.. -112
`MEMORY
`PROCEDURES i..-i-110
`DATA
`t--..~114
`
`v104
`
`RAID
`STORAGE
`SUB-SYSTEM
`(E.G. RAID 1 OR
`RAID O + 1)
`D D
`
`FIG. 25
`
`PayPal Ex. 1013, p. 18
`PayPal v. IOENGINE
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 18 of 19 US 2002/0004912 Al
`
`,-100
`v 101
`
`HOST COMPUTER
`
`HOST
`PROCESSOR
`
`102
`-
`
`BUSS
`
`I
`109.-/
`
`MEMORY ~103
`(E.G. RAM)
`t.-110
`PROCEDURES
`DATA
`
`114
`
`v104
`
`RAID
`STORAGE
`SUB-SYSTEM
`(E.G. RAID 1 OR
`RAID0 + 1)
`E3
`E3
`
`FIG. 26
`
`PayPal Ex. 1013, p. 19
`PayPal v. IOENGINE
`
`

`

`Patent Application Publication
`
`Jan. 10, 2002 Sheet 19 of 19 US 2002/0004912 Al
`
`RAID 1
`
`RAID 0+1
`
`..--
`----
`ABC
`DISK
`
`..___
`
`RAID SET
`
`--......
`
`~
`
`ABC
`DISK
`
`FIG. 27
`
`..---
`...___
`
`RAID SET
`
`___.,
`
`ABC
`DISK
`
`I
`
`DEF
`DISK
`
`..____
`
`ABC
`DISK
`'
`DEF
`DISK
`
`~
`
`FIG. 28
`
`PayPal Ex. 1013, p. 20
`PayPal v. IOENGINE
`
`

`

`US 2002/0004912 Al
`
`Jan. 10, 2002
`
`1
`
`SYSTEM, ARCHITECTURE, AND METHOD FOR
`LOGICAL SERVER AND OTHER NETWORK
`DEVICES IN A DYNAMICALLY CONFIGURABLE
`MULTI-SERVER NETWORK ENVIRONMENT
`
`RELATED APPLICATION
`
`[0001] This application is a continuing application under
`35 U.S.C. §§ 119(e) and 120, wherein applicant and inventor
`claim the benefit of priority to United States Provisional
`application Ser. No. 60/283,375 entitled System, Method
`And Architecture For Dynamic Server Power Management
`And Dynamic Workload Management for Multi-Server
`Environment filed Apr. 11, 2001; United States Provisional
`application Ser. No. 60/236,043 entitled System, Apparatus,
`and Method for Power-Conserving Multi-Node Server
`Architecture filed Sep. 27, 2000; and United States Provi(cid:173)
`sional Application Serial No. 60/236,062 entitled System,
`Apparatus, and Method for Power Conserving and Disc(cid:173)
`Drive Life Prolonging RAID Configuration filed Sep. 27,
`2000; each of which application is hereby incorporated by
`reference.
`
`[0002] The following United States Utility patent appli(cid:173)
`cations are also related applications: U.S. Utility patent
`application Ser. No. 09/___ (Attorney Docket No.
`A-70531/RMA) entitled System, Method, and Architecture
`for Dynamic Server Power Management and Dynamic
`Workload Management for Multi-server Environment filed
`May ___ 2001; U.S. Utility patent application Ser. No.
`09/ ___ (Attorney Docket No. A-70532/RMA) entitled
`System and Method for Activity or Event Based Dynamic
`Energy Conserving Server Reconfiguration filed __ _
`May 2001; U.S. Utility patent application Ser. No.
`09/ ___ (Attorney Docket No. A-70533/RMA) entitled
`System, Method, Architecture, and Computer Program
`Product for Dynamic Power Management in a Computer
`System filed May ___ 2001; U.S. Utility patent appli-
`cation Ser. No. 09/ ___ (Attorney Docket No. A-70534/
`RMA) entitled Apparatus, Architecture, and Method for
`Integrated Modular Server System Providing Dynamically
`Power-managed and Work-load Managed Network Devices
`filed May ___ 2001;U.S. Utility patent application Ser.
`No.09/__ (Attorney Docket No. A-70535/RMA)
`entitled System, Architecture, and Method for Logical
`Server and Other Network Devices in a Dynamically Con(cid:173)
`figurable Multi-server Network Environment
`filed
`May ___ 2001; U.S. Utility patent application Ser. No.
`09/ ___ (Attorney Docket No. A-70536/RMA) entitled
`Apparatus and Method for Modular Dynamically Power(cid:173)
`Managed Power Supply and Cooling System for Computer
`Systems, Server Applications, and Other Electronic Devices
`filed ___ May 2001; and, U.S. Utility patent application
`Ser. No. 09/ __ (Attorney Docket No. A-70537/RMA)
`entitled Power on Demand and Workload Management
`System and Method; each of which applications is hereby
`incorporated by reference.
`
`[0003] This is also a continuing application claiming the
`benefit of priority under 35 U.S.C. § 120 to each of the
`following applications: U.S. patent Application Ser. No.
`09/558,473 filed Apr. 25, 2000, entitled System and Method
`Of Computer Operating Mode Clock Control For Power
`Consumption Reduction; which is a continuation of U.S.
`patent application Ser. No. 09/121,352 filed Jul. 23, 1998,
`entitled System and Method of Computer Operating Mode
`
`Control for Power Consumption Reduction; which is a
`division of application Ser. No. 08/767,821 filed Dec. 17,
`1996, entitled Computer Activity Monitor Providing Idle
`Thread and Other Event Sensitive Clock and Power Control
`abandoned; which is a continuation of application Ser. No.
`08/460,191 filed Jun. 2, 1995, entitled Activity Monitor That
`Allows Activity Sensitive Reduced Power Operation of a
`Computer System abandoned; which is a continuation of
`application Ser. No. 08/285,169 filed Aug. 3, 1994, entitled
`Power Management for Data Processing System, aban(cid:173)
`doned; which is a continuation of application Ser. No.08/
`017,975 filed Feb. 12, 1993 entitled Power Conservation
`Apparatus Having Multiple Power Reduction Levels Depen(cid:173)
`dent Upon the Activity of a Computer System, U.S. Pat. No.
`5,396,635; which is a continuation of application Ser. No.
`07/908,533 filed Jun. 29, 1992 entitled Improved Power
`Management for Data Processing System, abandoned;
`which is a continuation of application Ser. No. 07/532,314
`filed Jun. 1, 1990 entitled, Power Management for Data
`Processing System, now abandoned; each of which appli(cid:173)
`cations are hereby incorporated by reference.
`
`[0004] This application is also related to: U.S. Pat. No.
`6,079,025 issued June 20, 2000 entitled system and Method
`of computer Operating mode Control For power Consump(cid:173)
`tion system; U.S. Pat. No. 5,892,959 issued Apr. 6, 1999
`entitled Computer Activity Monitor Providing Idle Thread
`And Other Event Sensitive Clock and Power Control; U.S.
`Pat. No. 5,799,198 issued Aug. 25, 1998 entitled Activity
`Monitor For Computer systems Power Management; U.S.
`Pat. No. 5,758,175 issued May 26, 1998 entitled Multi(cid:173)
`Mode Power Switching For Computer Systems; U.S. Pat.
`No. 5,710,929 issued Jan. 20, 1998 entitled Multi-State
`Power Management For Computer System; and U.S. Pat.
`No. 5,396,635 issued Mar. 7, 1995 for Power Conservation
`Apparatus Having Multiple Power Reduction Levels Depen(cid:173)
`dent Upon the Activity of a Computer System; each of which
`patents are herein incorporated by reference.
`
`FIELD OF THE INVENTION
`
`[0005] This invention pertains generally to architecture,
`apparatus, systems, methods, and computer programs and
`control mechanisms for managing power consumption and
`work-load in data and information servers; more particularly
`to power consumption and workload management and con(cid:173)
`trol systems for high-density multi-server computer system
`architectures that maintain performance while conserving
`energy and to the method for power management and
`workload management used therein, and most particularly to
`system, method, architectures, and computer programs for
`dynamic server power management and dynamic workload
`management for multi-server environments.
`
`BACKGROUND
`
`[0006] Heretofore, servers generally, and multi-node net(cid:173)
`work servers in particular, have paid little if any attention to
`power or energy conservation. Such servers were designed
`and constructed to run at or near maximum levels so as to
`serve data or other content as fast as possible, or where
`service demands were less than capacity to remain ever
`vigilant to provide fast response to service requests. Increas(cid:173)
`ing processor and memory speeds have typically been
`accompanied by higher processor core voltages to support
`the faster device switching times, and faster hard disk drives
`
`PayPal Ex. 1013, p. 21
`PayPal v. IOENGINE
`
`

`

`US 2002/0004912 Al
`
`Jan. 10, 2002
`
`2
`
`have typically lead to faster and more energy-hungry disk
`drive motors. Larger memories and caches have also lead to
`increased power consumption even for small single-node
`servers. Power conservation efforts have historically focused
`on the portable battery-powered notebook market where
`battery life is an important marketing and use characteristic.
`However, in the server area, little attention has been given to
`saving power, such servers usually not adopting or utilizing
`even the power conserving suspend, sleep, or hibernation
`states that are available with some Microsoft 95/98/2000,
`Linux, Unix, or other operating system based computers,
`personal computers, PDAs, or information appliances.
`
`[0007] Multi-node servers present a particular energy con(cid:173)
`sumption problem as they have conventionally be archi(cid:173)
`tected as a collection of large power hungry boxes intercon(cid:173)
`nected by external interconnect cables. Little attention has
`been placed on the size or form factor of such network
`architectures, the expansability of such networks, or on the
`problems associated with large network configurations.
`Such conventional networks have also by-and-large paid
`little attention to the large amounts of electrical power
`consumed by such configurations or in the savings possible.
`This has been due in part because of the rapid and unex(cid:173)
`pected expansion in the Internet and in servers connected
`with and serving to Internet clients. Internet service com(cid:173)
`panies and entrepreneurs have been more interested in a
`short time to market and profit than on the effect on electrical
`power consumption and electrical power utilities; however,
`continuing design and operation without due regard to
`power consumption in this manner is problematic.
`
`[0008] Networks servers have also by-and-large neglected
`to factor into the economics of running a network server
`system the physical plant cost associated with large rack
`mounted equipment carrying perhaps one network node per
`chassis. These physical plant and real estate costs also
`contribute to large operating costs.
`
`[0009]
`In the past, more attention was given to the pur(cid:173)
`chase price of equipment and little attention to the operating
`costs. It would be apparent to those making the calculation
`that operating costs may far exceed initial equipment pur(cid:173)
`chase price, yet little attention has been paid to this fact.
`More recently, the power available in the California electri(cid:173)
`cal market has been at crisis levels with available power
`reserves dropping below a few percent reserve and rolling
`blackouts occurring as electrical power requirements drop
`below available electrical power generation capacity. High
`technology companies in the heart of Silicon Valley cannot
`get enough electrical power to make or operate product, and
`server farms which consume vast quantities of electrical
`energy for the servers and for cooling equipment and facili(cid:173)
`ties in which they are housed, have stated that they may
`relocated to areas with stable supplies of low-cost electricity.
`
`[0010] Even were server manufactures motivated to adopt
`available power management techniques, such techniques
`represent only a partial solution. Conventional computer
`system power management tends to focus on power man(cid:173)
`aging a single CPU, such as by monitoring certain restricted
`aspects of the single CPU operation and making a decision
`that the CPU should be run faster to provide greater perfor(cid:173)
`mance or more slowly to reduce power consumption.
`
`[0011] Heretofore, computer systems generally, and server
`systems having a plurality of servers where each server
`includes at least one processor or central processing unit
`(CPU) in particular have not been power managed to main(cid:173)
`tain performance and reduce power consumption. Even
`where a server system having more than one server com(cid:173)
`ponent and CPU may possibly have utilized a conventional
`personal computer architecture that provided some measure
`of localized power management separately within each
`CPU, no global power management architecture or methods
`have conventionally been applied to power manage the set
`of servers and CPUs as a single entity.
`
`[0012] The common practice of over-provisioning a server
`system so as to be able to meet peak demands has meant that
`during long periods of time, individual servers are consum(cid:173)
`ing power and yet doing no useful work, or several servers
`are performing some tasks that could be performed by a
`single server at a fraction of the power consumption.
`
`[0013] Operating a plurality of servers, including their
`CPU, hard disk drive, power supply, cooling fans, and any
`other circuits or peripherals that are associated with the
`server, at such minimal loading also unnecessarily shortens
`their service life. However, conventional server systems do
`not consider the longevity of their components. To the extent
`that certain of the CPUs, hard disk drives, power supplies,
`and cooling fans may be operated at lower power levels or
`for mechanical systems (hard disk drive and cooling fans in
`particular) their effective service life may be extended.
`
`[0014] Therefore there remains a need for a network
`architecture and network operating method that provides
`large capacity and multiple network nodes or servers in a
`small physical footprint and that is power conservative
`relative to server performance and power consumed by the
`server, as well as power conservative from the standpoint of
`power for server facility air conditioning. These and other
`problems are solved by the inventive system, apparatus and
`method. There also remains a need for server farms that are
`power managed in an organized global manner so that
`performance is maintained while reducing power consump(cid:173)
`tion. There also remains a need to extend the effective
`lifetime of computer system components and servers so that
`the total cost of ownership is reduced.
`
`SUMMARY
`
`[0015] Aspects of the invention provide network architec(cid:173)
`ture, computer system and/or server, circuit, device, appa(cid:173)
`ratus, method, and computer program and control mecha(cid:173)
`nism for managing power consumption and workload in
`computer system and data and information servers. Further
`provides power and energy consumption and workload
`management and control systems and architectures for high(cid:173)
`density and modular multi-server computer systems that
`maintain performance while conserving energy and method
`for power management and workload management.
`Dynamic server power management and optional dynamic
`workload management for multi-server environments is
`provided by aspects of the invention. Modular network
`devices and integrated server system, including modular
`servers, management units, switches and switching fabrics,
`modular power supplies and modular fans and a special
`backplane architecture are provided as well as dynamically
`reconfigurable multi-purpose modules and servers.
`
`PayPal Ex. 1013, p. 22
`PayPal v. IOENGINE
`
`

`

`US 2002/0004912 Al
`
`Jan. 10, 2002
`
`3
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`[0016] FIG. 1 is a diagrammatic illustration showing a
`exemplary embodiment of an inventive power conserving
`high-density server system.
`[0017] FIG. 2 is a diagrammatic illustration showing an
`exemplary embodiment of a single 2U high rack mountable
`Integrated Server System Unit having a plurality of modular
`server units.
`[0018] FIG. 3 is a diagrammatic illustration showing a
`standard server farm architecture in which multiple nodes
`are individually connected by cables to each other to form
`the desired network.
`[0019] FIG. 4 is a diagrammatic illustration showing an
`embodiment of the inventive Integrated Appliance Server
`(IAS) standard architecture also or alternatively referred to
`as an Integrated Server System (ISS) architecture in which
`multiple nodes selected from at least a computer node (CN)
`such as a server module (SM), network node (NN) also
`referred to as a switch module, and monitor or management
`node (MN) also referred to as a Management Module (MM)
`are provided within a common enclosure and coupled
`together via an internal backplane bus.
`[0020] FIG. 5 is a diagrammatic illustration showing
`another embodiment of the invention in which multiple
`modular IAS ( or ISS) clusters each containing multiple
`nodes are cascaded to define a specialized system.
`[0021] FIG. 6 is a diagrammatic illustration showing an
`embodiment of an Integrated Server System Architecture
`having two interconnected integrated server system units
`(ISSUs) and their connectivity with the external world.
`[0022] FIG. 7 is a diagrammatic illustration showing an
`exemplary embodiment of an AMPC bus and the connec(cid:173)
`tivity of Server Modules and Management Modules to the
`bus to support serial data, video, keyboard, mouse, and other
`communication among and between the modules.
`[0023] FIG. 8 is a diagrammatic illustration showing an
`exemplary embodiment of ISSU connectivity to gigabit
`switches, routers, load balances, and a network.
`[0024] FIG. 9 is a diagrammatic illustration showing an
`embodiment of the inventive power conserving power man(cid:173)
`agement between two servers and a manager.
`[0025] FIG. 10 is a diagrammatic illustration showing an
`alternative embodiment of a server system showing detail as
`to how activity may be detected and operating mode and
`power consumption controlled in response.
`[0026] FIG. 11 is a diagrammatic illustration showing
`another alternative embodiment of a server system particular
`adapted for a Transmeta Crusoe™ type processor having
`LongRun™ features showing detail as to how activity may
`be detected and operating mode and power consumption
`controlled in response.
`[0027] FIG. 12 is a diagrammatic illustration showing
`aspects of the connectivity of two management modules to
`a plurality of server modules and two Ethernet switch
`modules.
`[0028] FIG. 13 is a diagrammatic illustration showing an
`exemplary internetwork and the manner in which two dif(cid:173)
`ferent types of master may be deployed to power manage
`such system.
`
`[0029] FIG. 14 is a diagrammatic illustration showing a
`graph of the CPU utilization (processor activity) as a func(cid:173)
`tion of time, wherein the CPU utilization is altered by
`entering different operating modes.
`[0030] FIG. 15 is a diagrammatic illustration showing an
`exemplary state engine