(12) United States Patent
`Lo et al.
`
`I 1111111111111111 11111 111111111111111 1111111111 1111111111111111 IIII IIII IIII
`US006877044 B2
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,877,044 B2
`Apr. 5, 2005
`
`(54) DISTRIBUTED STORAGE MANAGEMENT
`PLATFORM ARCHITECTURE
`
`(75)
`
`Inventors: Horatio Lo, Milpitas, CA (US); Sam
`Tam, Belmont, CA (US); David Lee,
`San Jose, CA (US); Dietmar M.
`Kurpanek, Emerald Hills, CA (US)
`
`(73)
`
`Assignee: Vicom Systems, Inc., Fremont, CA
`(US)
`
`( *)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 77 days.
`
`(21) Appl. No.: 09/904,824
`
`(22) Filed:
`
`Jul. 12, 2001
`
`(65)
`
`Prior Publication Data
`
`US 2002/0103943 Al Aug. 1, 2002
`
`Related U.S. Application Data
`
`( 63) Continuation-in-part of application No. 09/501,881, filed on
`Feb. 10, 2000, now Pat. No. 6,772,270.
`(60) Provisional application No. 60/268,777, filed on Feb. 14,
`2001, and provisional application No. 60/217,867, filed on
`Jul. 12, 2000.
`Int. Cl.7 ................................................ G06F 13/00
`
`(51)
`
`(52) U.S. Cl. .............................. 710/2; 710/38; 710/313
`(58) Field of Search .............................. 710/1-2, 8, 29,
`710/36-38, 51, 62-64, 72-74, 305-317;
`711/100, 114; 709/249; 714/7, 48; 716/2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,745,701 A * 4/1998 Nguyen-Thai et al. ...... 709/249
`5,768,623 A * 6/1998 Judd et al. .................... 710/37
`6,128,750 A * 10/2000 Espy et al.
`.................... 714/7
`6,233,702 Bl * 5/2001 Horst et al. ................... 714/48
`* cited by examiner
`
`Primary Examiner-Christopher B. Shin
`(74) Attorney, Agent, or Firm-Campbell Stephenson
`Ascolese LLP; Samuel G. Campbell, III
`
`(57)
`
`ABSTRACT
`
`A distributed storage management platform (DSMP) archi(cid:173)
`tecture is disclosed. Such a DSMP architecture includes a
`number of storage routers. Each one of the storage routers
`comprises a number of interface controllers. One of the
`interface controllers of each one of the storage routers is
`communicatively coupled to one of the interface controllers
`of at least one other of the storage routers.
`
`38 Claims, 18 Drawing Sheets
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 1
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 1 of 18
`
`US 6,877,044 B2
`
`Host
`12.QU}
`
`FIG. 1A
`
`Hos\ 1/0
`Interface
`Controller
`illill
`
`Port 126(1)
`
`L.---/
`
`[=:J
`Port
`
`Hos
`lnte
`Con
`
`Storage Sid
`Interface
`Controller
`
`JI
`
`c=J
`Port
`
`Ho'
`lnte
`Con
`
`Storage Sid
`Interface
`Controller
`
`Port
`
`c:=i
`
`I
`c=J
`II
`
`Host 1/0
`Interface
`Controller
`125(2}
`
`Host
`120(2)
`
`Port 126(2)
`
`9
`
`Port
`102(1, 1)
`
`. . .
`
`Host Side
`Interface Controller
`1Q1fil
`
`9
`
`Pod
`102(i,N)
`
`Storage Side
`Interface
`Controller
`105(i}
`
`Port 106(i,N)
`c::Jl
`Controller
`
`c=J
`
`I
`
`I
`
`c:=i
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`
`-------------------::-=:>
`
`~·5a-5a
`
`Storage Drives Array/Subsystem
`LlPill
`
`-------------------------
`
`·· 5a 5a ·@
`
`Storage Drives Arroy/Subsr-,tem
`130(2)
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 2
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 2 of 18
`
`US 6,877,044 B2
`
`. . .
`
`Host 1/0
`Interface
`Controller
`125(N)
`
`Hosl
`120(N}
`
`Port 126(N)
`
`FIG. 1A FIG. 18
`
`Key To
`FIG. 1
`
`>tern
`ace
`ce
`
`I
`
`system
`erface
`evice
`
`system 7
`-
`
`rfoce
`VI Ce
`
`~yslem
`erfoce
`evice
`
`""}...,''"''
`erface
`evice
`
`SAN
`l1Q
`
`v • - - -
`
`Subsystem
`lntedoce
`Device
`100(i)
`
`Port
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`
`- @_@ ___ @
`
`----------------------..
`
`Storage Drives Array/Subsystem
`llQfrj}
`
`FIG. 1B
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 3
`
`

`

`N
`~
`,I;;..
`,I;;..
`b
`-...,l
`-...,l
`~
`_,.a-...
`rJ'J.
`
`e
`
`~
`
`'"""' 00
`0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`Ul
`0
`0
`N
`~Ul
`
`~ :;
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FIG. 2
`
`22B
`
`Card
`
`Interface
`Network
`Host
`
`224
`ROM & RAM
`System Memory-
`
`22J
`Chip-set
`
`290
`Electrical Hardware
`
`& Other
`
`Power Supply
`
`222
`
`CPU
`
`221
`Mother-boord
`
`225
`
`Controller
`Interface
`Host 1/0
`
`V PCI Bus 270
`
`220
`
`H ,st
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 4
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 4 of 18
`
`US 6,877,044 B2
`
`Host 1/0
`Interface
`Controller
`325(1)
`
`Host
`320(1)
`
`Port
`326(1)
`Terminal
`Terminal
`327(1,oul) _, __ ~
`- ~ 327(1.in)
`
`315
`r-------------------- --------------------------------------------- )------------~
`
`314-
`
`-
`
`i---- 311
`
`In-Band SAN Links
`
`I
`
`I
`
`I
`
`312-
`
`t
`313
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`L _________ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ,
`
`/~ Interface Controller
`Part
`30l(l)
`302(1,1)\
`
`~-Terminal
`Port
`303(1,N,in)
`302(1,N)
`
`\
`l erminal
`Terminal
`303(1,1,out) 303(1,1,in)
`
`Terminal
`303(1,N,out)
`
`Subsystem
`Interface Device
`300(1)
`
`1
`
`I
`
`!
`' ' '
`
`I
`I
`
`I
`
`LL . ~-@:~--§
`
`Storage Drives Array/Subsystem
`330(1)
`
`FIG. 3A
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 5
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 5 of 18
`
`US 6,877,044 B2
`
`Hosl 1/0
`Interface
`Controller
`325(2)
`
`Port
`326(2)
`
`Host
`320(2)
`
`Terminal
`327(2,in)
`
`Terminal
`327(2,out)
`
`I 31
` !
`------------------------------------------------- ------ --cIJ
`313 ~ I
`
`I
`
`15
`
`315
`
`312
`
`lnterf ace Controller
`301(2)
`
`Terminal
`303(2,N,in)
`
`I
`
`L------------------
`
`Terminal
`303(2, 1,out)
`
`Terminal
`303(2, 1,in)
`
`Terminol
`303(2,N,oul)
`
`Subsystem
`Interface Device
`300(2)
`
`I
`I
`r----------- ------------------------------------------------------------------------------1
`
`Storage Drives Array/Subsystem
`330(2)
`
`FIG. 3B
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 6
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 6 of 18
`
`US 6,877,044 B2
`
`Host 1/0
`Interface
`Controller
`325(N)
`
`Port
`326(N)
`
`Host
`320(N)
`
`Terminal
`327(N,in)
`
`Terminal
`327(N,out)
`
`~
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`-
`I
`~
`I
`L------------------------ - ---------------------------------------7
`I
`SAN --J
`'
`310
`
`FIG. 3A FIG. 38 FIG. 3C
`
`~-~-~-~
`
`Key To
`FIG. 3
`
`I
`I
`
`312
`
`Interface Controller
`301(N}
`
`Port
`302(N,N)
`
`Terminal
`303(N,N,in)
`
`Terminal
`303(N,1,oul)
`
`Terminal
`303{N,1,in)
`
`Terminal
`303(N,N,out)
`
`Subsystem
`Interface Device
`300(N)
`
`' ' ' I
`' I
`
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`:
`I
`r----------- ------------------------------------------------------~
`
`@ili=:=§
`
`Storage Drives Array/Subsystem
`330(N)
`
`FIG. 3C
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 7
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 7 of 18
`
`US 6,877,044 B2
`
`Host 1/0
`Interface
`Controller
`12-5.ill
`
`Host
`420(1)
`
`Host Network
`Interface
`Card
`428( I}
`
`I
`
`Terminal
`27(1,out)
`4
`
`Port
`426(1)
`'t!~ I!:.)
`
`Terminal
`- 427(1.in)
`
`Port 429(1)
`
`?
`
`415
`----------------- ·- >---------------------- ---- ---- ~-- -- -------- -;---- ------ ------~
`
`In-Band
`SAN Links
`
`- - - - - --
`
`~ - - - - - - - - - - - - - - - - -
`
`414- -
`(I/Port
`
`/.__:]~
`
`402( 1, 1)
`
`-411
`
`SAN Links
`
`Interface Controller
`.4filill
`
`I
`
`Port
`402(1,N)
`
`Terminal
`403(1,1,out)
`
`'Terminal
`403(1,1,in)
`
`/
`
`Terminal
`403(1,N,out)
`
`I r~
`
`,- Terminal
`403( 1,N,in)
`
`I
`L _ _ _ _ _ _ _ _ _ _ _ _ _ _ I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`412
`I
`
`(
`413
`
`,--(cid:173)
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`r
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`L - -1--D Port 409( 1)
`
`Subsystem 1/F Device
`Network Interface Card
`408(1)
`
`Subsystem
`Inter! ace Device
`400(1)
`
`t
`I
`I
`------- -----------------------------------------------------------------------4
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`Storage Drives Array /Subsystem
`430(1)
`
`FIG. 4A
`
`L ___ ------------------------·-:::::,
`'
`
`@@----~
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 8
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 8 of 18
`
`US 6,877,044 B2
`
`Host 1/0
`Interface
`Controller
`425(2)
`
`Host
`420(2)
`
`Host Nelwork
`Interface
`Card
`428(2)
`
`Port 426(2)
`Terminal
`..----- 427(2,in)
`,,
`
`Par\ 429(2)
`~
`
`: Terminal
`1 427(2,oul) ----
`
`~ -- -------------=-- ------------ -- --- ---7---- ----------'. /
`
`I
`I
`I
`I
`I
`I
`I
`I
`:
`
`1
`
`I
`
`•
`
`•
`
`~ L1J
`
`•
`
`5
`--
`
`I
`
`- - - - - - - - •-
`I
`
`- - - - - 7
`
`, - - - - _J
`
`441
`r----- _______ .J
`
`L ______ _
`
`412
`
`Out-Of-Bond
`
`In-Band
`I
`SAN Links 1
`I
`- - - - - - - - - - - - - - - I
`441
`I
`I
`-----------------7
`
`rn I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`L
`
`Interface Controller
`401(2)
`
`413
`
`Terminal
`403( 2,N, in)
`
`Terminal
`403(2, 1,out)
`
`Terminal
`403(2,1,in)
`
`Terminal
`403(2,N,oul)
`
`- +o Port 409(2)
`
`Subsystem 1/F Device
`Network Interface Cord
`408(2)
`
`Subsystem
`In terf oce Device
`400(2)
`
`I
`
`~------------- ----------------------------------------------------------------------------~
`
`I
`I
`
`------------------------~
`
`-~-5a----~
`
`Storage Drives Arroy/Subsrtem
`430(2)
`
`. . .
`
`FIG. 4B
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 9
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 9 of 18
`
`US 6,877,044 B2
`
`Host
`420(N)
`
`Host 1/0
`Interface
`Controller
`425(N)
`
`Terminal
`427(N,out)
`
`Terminal
`Porl 426(N)
`__________,_ 427(N,in)
`
`Host Network
`Interface
`Cord
`428(N)
`
`Port 429(N)
`
`FIG. 4A FIG. 4B FIG. 4C
`
`Key To
`FIG. 4
`
`-----------------------------~------------------,
`i
`I
`i
`I
`SAN --l
`1
`i
`I
`410
`i
`I
`I
`i
`I
`j
`:
`~-------------------------J
`I
`I
`I ,-7
`
`SAN Links
`
`412
`
`I
`I
`I
`
`Interface Controller
`401(N)
`
`Terminal
`403(N,N,in)
`
`Port
`402(N,N)
`
`Terminal
`403(N,1,out)
`
`Terminal
`403(N,1,in)
`
`Terminal
`403(N,N,out)
`
`L
`
`--f-D Port 409(N}
`
`Subsystem I /F Device
`Network Interface Card
`408{N)
`
`Subsystem
`Interface Device
`400(N)
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`1
`I
`I
`
`~----------------------------------------------------------------------I
`
`--------------------------:::)
`
`-~@------@
`
`Storage Drives Array /Subsystem
`430(N)
`
`FIG. 4C
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 10
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 10 of 18
`
`US 6,877,044 B2
`
`Host Fibre
`Channel Interface
`Controller
`525( 1)
`
`Host
`520(1)
`
`I
`I
`I
`I
`I
`I
`I
`I
`Port
`I
`526(1)
`Terminal
`Terminal
`I
`5 27(1,out) - ~~ - 527(1,in)
`I
`515
`I
`---------------- •- -----------------------------------------------1------------j
`
`,---
`
`' I
`
`I
`I
`I
`
`\
`
`r---511
`
`e -
`
`In-Band SAN Links
`
`(
`514
`
`512
`l
`
`(
`
`,_ Terminal
`503(1,N,in)
`
`~ 513
`
`I
`I
`
`Port
`502(1,N)
`
`Terminal
`503( 1, 1 ,out)
`t
`--
`Host Side Multi-Port
`~
`Fibre Channel Controller
`Port
`lillill
`502(1,1)\
`
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`I
`!
`I
`I
`'
`
`I
`Terminal
`503(1,1,in)
`
`Termi/(cid:143) 1
`503(1,N,out)
`
`Storage
`Router
`500(1)
`
`Terminal
`507(1,1,in) -
`
`I
`
`516
`l
`
`Device Side Multi-Port
`Port
`Port
`.
`506(l,1) Fibre Cha;;;(l)ontroller SOfi(l,N)
`Terminal
`~ ~ ~-~ - 507(1JJ,ou\)
`-
`I
`I
`Te;minal
`Terminal_; 517~
`507( 1, 1,out) 507(1,N,in)
`
`521-
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`Terminal
`533( 1,in)
`
`TOO
`
`TOI
`
`TO ... x
`
`Storage Drives Array/Subsystem
`530(1)
`
`·
`
`FIG. SA
`
`519
`516~
`I
`I ..._ __
`------ - ----------------------------------------------------------------------~
`- ~ ~-Terminal
`-®@ _______
`LC== Port 532(1) _ SJJ(l,oul) ______________ ~ I
`Storage
`Subsystem Fibre
`Channel Controller
`~ll
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 11
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 11 of 18
`
`US 6,877,044 B2
`
`Host
`520(2)
`
`Host Fibre
`Channel Interface
`Controller
`525(2)
`
`Terminal
`527(2,out)
`
`Port
`526(2)
`
`Terminal
`527(2,in)
`I
`I 5~5
`!
`------------------------------------------------- ------ --t=U
`
`515
`
`In Bond SAN Links
`
`512
`
`Host Side Multi-Port
`Fibre Channel Controller
`501(2)
`
`r.Jii~--L..Terminol
`503(2,N,in)
`
`513
`
`I
`I
`
`~ I
`
`Terminal
`503(2,1,in)
`
`Terminal
`503( 2,N,out)
`
`I
`~------------------
`
`Terminal
`503(2,1,out)
`
`Storage
`Router
`500(2)
`
`Terminal
`507(2,1,aut)
`
`Device Side Multi-Port
`Fibre Channel Controller
`505(2)
`
`Port )
`506(2,N
`
`Terminal
`507(2,1,in) ~~
`'-------+-+---------+-I-----'
`
`Terminal
`507( 2,N,in)
`
`517
`
`Terminal
`507(2,N,out)
`
`516
`
`I
`
`li1: I
`
`I
`---------------------------------------------------
`~---------------•
`I
`.
`( ~~ Port 532(2)
`Terminal
`I
`,
`533(2,out)
`I
`I ;;;;;;)
`
`.
`
`:~bs~;;:;'r::" _, -~ -~ ·ccc c-§
`
`Channel Controller
`SJl(Z)
`
`.
`Storage Dnves Array/Subsystem
`530(2)
`
`I
`I
`I
`I
`I
`I
`
`I
`
`( --- --- ~-- (---~
`519
`519
`I
`1
`I
`I
`I
`I
`I
`I
`I
`I
`
`. . .
`
`FIG. SB
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 12
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 12 of 18
`
`US 6,877,044 B2
`
`Host Fibre
`Channel Interface
`Controller
`~
`
`Host
`520(N)
`
`FIG. SA FIG. 58 FIG. SC
`
`____ _._ _ _.____~
`
`Key To
`FIG. 5
`
`Port
`526(N)
`
`Terminal
`527(N;in)
`
`Terminal
`527(N,out) --.i_i__...---
`L------------------------ - -------------------------------------------------, I
`SAN ---1
`510
`
`I
`I
`
`L,__, __ __,
`
`Host Side Multi-Port
`Fibre Channel Controller
`501(N)
`
`,--='.'....!.__,_ Terminal
`503(N,N,in)
`
`Terminal
`503(N,1,out)
`
`Terminal
`503(N,1,in)
`
`Terminal
`503(N,N,out)
`
`Storage
`Router
`SOO(N}
`
`Terminal
`507(N,1,in)
`
`Terminal
`507(N, 1,oul)
`
`Port
`506(N, 1)
`
`Device Side Multi-Port
`Fibre Channel Controller
`SOS(N)
`
`Port
`506(N,N)
`
`Terminal
`507{N,N,out)
`
`517
`
`Terminal
`507(N,N,in)
`
`I
`I
`I
`I
`I
`I
`I
`
`L-- ..
`I
`
`Terminal
`1 533(N,in)
`
`In-Bond SAN Links
`-------- - -------- ---------------------------------------------------~
`Port 532(N)
`
`Terminal
`533(N,out)
`
`:::,~;:~•Fb,e :__-6~~f@:=:~~
`
`Channel Controller
`531 (N)
`
`.
`Storage Onves Array/Subsystem
`530(N)
`
`FIG. 5C
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 13
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 13 of 18
`
`US 6,877,044 B2
`
`Host
`620(1)
`
`Host Fibre
`Channel Interface
`Controller
`625(1)
`
`Host Network
`Interface
`Cord
`628(1)
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`r - -
`1
`I
`
`I ,____
`I ' I
`I ' I
`
`I
`I
`
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`Terminal
`603(1,1,out)
`
`\
`
`-611
`Terminal
`,,603(1,1,in)
`
`In-Band
`SAN Links
`
`I
`614
`
`612
`1
`Terminal
`603( 1,2,ou\) .__
`(
`~ 613
`- Terminal
`Port
`603(1,2,in)
`602(1,2)
`
`I
`\
`\
`1
`~ Host Side Dual Port
`Fibre Channel Controller
`Port
`§_Qjfil
`602(1,1)
`
`I
`
`SVE
`Device
`600(1)
`
`SVE Device Network
`Interface Card
`608(1)
`
`Port 609(1) o-- _____ _j
`
`Port 629(1) 0- ... 1------7
`I
`I
`I
`I
`I
`Port
`I
`I
`626(1)
`Terminal
`Terminal
`I
`-
`I
`627(1,in)
`27(1,out)
`6
`t! ;!.!:c!:.J
`I
`515
`I
`---------------- - -----------------------------------------------1----T _______ j
`SAN 610
`I
`L ____ !
`-
`r ___ J
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`Port
`
`606(1,l) Fibre Ch~;;;1
`
`Terminal
`607(1,1,in) -
`
`Device Side Dual Port
`Port
`.
`;onlroller 606(l,Z)
`Terminal
`~ ~ - 607(1,2,out)
`I
`Te~minal
`Terminal .J 617-
`607(1,1,out) 607(1,2,in)
`
`i-----
`
`I
`
`616
`1
`
`621~
`
`619
`616~
`l
`------ - ______________________________________________________________________ j
`-
`-Terminal
`~ ~Port 632()f
`633(1,out)
`Storage
`Subsystem Fibre
`Channel Controller
`ru(l}
`
`Terminal
`633(1,in)
`
`I
`
`'
`
`L.:: . : :.::-: : ==::... ~-·-··-··-··-··-··~ @
`~0·-··-··
`
`TO ... x
`T01
`TOO
`Storage Drives Array/Subsystem
`630(1)
`
`FIG. 6A
`
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 14
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 14 of 18
`
`US 6,877,044 B2
`
`Host
`620(2)
`
`Host Fibre
`Channel Interface
`Controller
`625(2)
`
`Terminal
`627(2,out)
`
`Port
`626(2)
`
`L------------------
`
`Terminal
`627(2in)
`
`Port 629(2)
`
`7
`I
`I
`I
`
`I
`
`Host Network
`l
`Interlace
`Card
`l
`628(2)
`~ - - -~ I
`
`O
`
`O
`
`O
`
`1
`
`I
`
`.
`
`------------' ---------------------------------L 61~ ::-_:--klJ
`I 1 6:3 l
`
`6-1
`
`L--------------------------------------------~
`2
`-+ - -+ - - - - - -~ ~IJ
`Terminol
`603(2,2,in)
`
`~----•------------
`Host Side Dual Port
`Fibre Channel Controller
`601(2)
`
`Terminal
`603(2,1,out)
`
`1
`1
`
`i
`I
`I
`I
`I
`I
`I
`I
`
`•
`
`•
`
`•
`
`1
`
`•
`
`•
`
`•
`
`Terminal
`603(2,1,in)
`
`Terminal
`603(2,2,out)
`
`SVE Device Network
`Interface Cord
`608(2)
`
`Port 609(2) o-.. -
`
`SVE
`Device
`_600(2)
`
`Terminal
`607(2,1,out)
`
`Terminal
`607(2,1,in)
`
`Device Side Dual Port
`Fibre Channel Controller
`605(2)
`
`Terminal
`607(2,2,in)
`
`617
`
`Terminal
`607(2,2,out)
`
`I
`
`616 ~ I
`
`I
`--------------------------------------------------------!.----- --- ----{---.!
`L-------------
`I
`619
`619
`rE 5r !'.II
`:---_ Terminal
`I
`f-
`I
`I
`(
`Port 632(2)
`633(2,out)
`I
`!
`! Terminal
`l___ '---------- ------------------------::::,
`I
`S"bs~::~•fibce G~f@§
`633(2,in)
`I
`I
`I
`I
`I
`I
`
`Channel Controller
`631 (Z)
`
`.
`Storage Drives Array/Subsystem
`630(2)
`
`. . .
`
`FIG. 6B
`
`I
`
`I
`
`i
`I
`'
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 15
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 15 of 18
`
`US 6,877,044 B2
`
`Host
`620(N)
`
`Host Fibre
`Channel Interface
`Controller
`625(N)
`Terminal
`,,.----____ 627(N,out)
`
`Terminal
`627(N,in)
`
`641
`
`Port 629(N)
`
`Host Netwcrk
`Interface
`Card
`628{N)
`
`FIG. 6A FIG. 6B FIG. 6C
`
`Key To
`FIG. 6
`
`--,
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`1
`
`~,
`
`_____ 2....1
`
`I
`:
`
`-------------------------------i-----------~
`Ou!-01-Band
`- - -~
`I
`1
`!
`_____________ 8 -Ne lwo r k Links
`:
`!
`Network
`L ___________ -
`-
`~------------ --
`:
`I
`.
`612
`640
`1...--------- 7
`I
`I ~ - - - - - - - - - -~
`l
`Host Side Dual Port
`/t:4'-=1E!J'~1 Terminal
`--,-,,..C-''"'""''
`Fibre Channel Controller
`603(N,2,in)
`601(N)
`
`1
`1
`I
`I
`I
`I
`I
`I
`I
`I
`....J
`
`I
`:
`
`:
`
`:
`:
`i
`i
`i
`i
`i
`i
`i
`:
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`r
`
`Terminal
`603(N, 1,ou!)
`
`Terminal
`603(N,1,in)
`
`Terminal
`603(N,2,out)
`
`SVE
`Device
`600(N)
`
`Terminal
`507(N,1,out)
`
`Terminal
`607(N,1,in)
`
`SVE Device Network
`Interface Card
`608(N}
`
`Port 609(N)
`
`Device Side Dual Port
`Fibre Channel Controller
`605(N)
`
`Terminal
`607(N,2,in)
`
`Terminal
`607(N,2,oul)
`
`1
`
`!
`
`I
`I
`I
`I
`I
`
`I
`
`:
`In-Bond
`:
`SAN Links
`I
`I
`l-------------- - ____________________________________________________________ J
`I
`,-- t!J
`( ) ...._ Terminal
`~ " Port 632 N
`633(N,out)
`
`I
`I
`/
`I
`I Terminal
`I 633(N,in)
`I
`I
`I
`I
`r
`
`Seb:~:~~efiboe -® @ -- _§-,
`
`Channel Controller
`§..Jl(_ti}
`
`Storage Drives Array/Subsystem
`630(N)
`
`FIG. 6C
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 16
`
`

`

`N
`~
`,I;;..
`,I;;..
`b
`-...,l
`-...,l
`~
`_,.a-...
`rJ'J.
`e
`
`'"""' 00
`0 ....,
`'"""' O'I
`~ ....
`'JJ. =(cid:173)~
`
`Ul
`0
`0
`N
`Ul
`"Cl :;
`>
`
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FIG. 7
`
`760
`Motherboard
`
`774
`
`SO RAM
`
`771b
`
`FPGA
`
`Sec on dory PCI
`
`771a
`
`FPGA
`
`Primary PCI
`
`785
`RS232 Serial Port
`
`778
`NV RAM
`
`777
`Memory
`Flash
`
`772
`PCI Arbiter
`
`FPGA
`
`Local Bus
`
`775
`
`PROM
`
`776
`SS RAM
`
`773
`
`Processor
`System
`
`Ethernet Port r---------------"'...._ __ .....,_ __ ----,1
`
`780
`
`-----------------------•----------------------,
`
`' I
`
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`' I
`
`I
`I
`I
`I
`I
`
`Connector
`FC Port
`
`Circuit
`
`FC Port By;iass
`
`Transceiver 763 ,----,
`
`761
`
`Controller
`Channel
`Fibre
`
`762
`
`SS RAM
`
`751 Doughterboard
`
`750
`
`Controller
`Channel
`Fibre
`
`752
`
`SS RAM
`
`Transceiver Jjl ---
`
`DD
`755(2)~~
`
`Connector
`FC Port
`
`1755(1)
`
`0
`
`
`1(cid:143)
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 17
`
`

`

`U.S. Patent
`
`Apr. 5, 2005
`
`Sheet 17 of 18
`
`US 6,877,044 B2
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 18
`
`

`

`N
`~
`,I;;..
`,I;;..
`Q
`_,.""-l
`-...,l
`~
`_,.a-...
`rJ'J.
`e
`
`'"""' 00
`0 ...,
`'"""' 00
`~ ....
`'JJ. =(cid:173)~
`
`Ul
`0
`0
`N
`~Ul
`
`t :;
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FIG. 9A
`
`930a
`Storage
`
`rnililillllllll II
`illllllillIIIll
`rnilllllllllllD ~ rnTIIlilIIIlllI
`l1lllllDillllll
`rnlllllillilll!
`l1lllllDillllll
`Olilililililil!
`l1lllilllll1lll
`Olllllilllllll!
`[Il[!]][[IIl]II
`rnililililillll
`
`rnDllilllllll!l /
`ffiUlllllllllll
`illilllililllll
`[]llil]l!Il]]]l
`ITlllllllilllll
`
`!Olilillllllllll
`(llill]Il]]ll]I
`Olilillllllllll
`Olilillllllllll
`illilllllllilII
`ffiIIIIIlllllll
`
`BBB B
`//\~~
`
`~=-= ~,,..,
`
`/
`
`/
`
`/
`
`/
`
`~,;;;;;-,;;;;-
`
`Il O O O (cid:143) a a(cid:143) ITU]]]]F Switch
`
`9990
`Data
`
`~Be] ~
`
`/
`
`I
`
`\
`
`'
`
`920a
`Ser·,ers
`
`I a·
`
`/
`
`illilllllililli1;
`rnnnnnnnrn
`rnilllllllillll
`rnilllllllillll
`rnDllllilllllll
`aillllillIIlilD
`
`930b
`Storage
`
`FIG. 9B
`
`/
`
`aillllllllIIIll11
`[Il]lll]IllllJl
`illllilllllil1l
`[I1]l]l!!Il]Ill
`illllilllllil1l
`illllilllllil1l
`I
`I
`/
`
`/
`
`!Illillllllilll
`flillilllilllll
`IDllllllililill
`!Illillllllilll
`(IlilJJjj]]Ilff
`[]]Il]J]]1]J]l]
`
`Meta Data
`
`992b
`Server
`
`I Servers I
`
`920b
`
`920b
`Servers I
`
`Meta Data
`
`995b
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 19
`
`

`

`US 6,877,044 B2
`
`1
`DISTRIBUTED STORAGE MANAGEMENT
`PLATFORM ARCHITECTURE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation-in-part of patent appli(cid:173)
`cation Ser. No. 09/501,881 filed on Feb. 10, 2000, now U.S.
`Pat. No. 6,772,270, entitled "A MULTI-PORT FIBRE
`CHANNEL CONTROLLER", referred to herein as MPFCC
`patent application, and having D. Kurpanek as the inventor,
`which is assigned to Vicom Systems, Inc., the assignee of the
`present invention, and which is hereby incorporated by
`reference herein, in its entirety and for all purposes.
`This application also claims priority to Provisional Patent
`Application Ser. No. 60/217,867, filed on Jul. 12, 2000,
`entitled "A DISTRIBUTED STORAGE MANAGEMENT
`PLATFORM ARCHITECTURE," and having H. Lo and S.
`Tam as inventors; and to Provisional Patent Application Ser.
`No. 60/268,777, filed on Feb. 14, 2000, entitled "A DIS(cid:173)
`TRIBUTED STORAGE MANAGEMENT PLATFORM
`ARCHITECTURE," also having H. Lo and S. Tam as
`inventors, which provisional patent applications are
`assigned to Vicom Systems, Inc., the assignee of the present
`application, and are hereby incorporated by reference herein, 25
`in their entirety and for all purposes.
`
`BACKGROUND OF THE INVENTION
`
`2
`properly set up and configured by an administrator, releases
`the end-user from the need to be concerned with the actual
`physical layout of the storage.
`Traditional approaches to storage and storage manage-
`s ment emphasize the need to control expense and restrict
`access. Prior technology has generally only enabled a given
`storage device to be accessible to a single server, so the latter
`goal is managed with relative ease though apparently work(cid:173)
`ing against achieving the former where there is such a
`10 one-to-one ratio of dependency. While one cost factor, that
`of the physical hardware performing the storage role, has
`exponentially decreased over the time since the advent of
`computing as a viable business tool, the cost associated with
`management now has continued to increase at an increasing
`15 rate, offsetting any benefits the former brings.
`The important function of managing storage in a net(cid:173)
`worked environment has proven to be generally difficult, and
`when comparing the approaches of management through a
`server-provided interface with storage-specific direct
`20 connect-based management, a definite trend for success has
`been correlated with the latter. Just as data has been seen to
`come to be valued as an independent strategic asset from the
`computers that access it, storage networking products and
`architectures, as platforms for data protection and storage
`management are just now being elevated to the same level
`of concern.
`To ensure reliability, redundant access is often supported,
`employing multiple routing devices, some of which may be
`30 distributed across geographically distant locations. Although
`a comparatively new technology, common existing imple(cid:173)
`mentations of SANs have to date been observed as failing in
`a critical area, that of not readily supporting extensibility. An
`essential characteristic for a SAN is that it must be scalable
`if it is to support the increasing rate of growth of demand for
`storage space.
`It is therefore desirable to introduce greater simplicity into
`the hardware used to communicate between a host system
`and storage array, while meeting the prerequisites of redun(cid:173)
`dancy and reliability ( collectively termed as high
`availability). Preferably, a suitable SAN architecture also
`provides improved performance and reduces the running
`cost of the SAN, ideally maintaining transparency to the
`user. Also most preferably, such an architecture is extensible,
`allowing easy insertion to and removal from the SAN of
`hosts, storage drive arrays or subsystems, and any appli-
`ances that are introduced into the SAN to form part of that
`architecture.
`
`35
`
`1. Field of the Art
`The present invention relates to computer subsystem
`communications, and, more particularly, to a method and
`apparatus for creating an extensible storage area network
`(SAN) architecture by interconnecting multiple storage
`router type devices-which may be viewed as the nodes in
`a framework which is the SAN-providing coupling of host
`computers to storage drive arrays or subsystems.
`2. Description of the Related Art
`Although most of the concepts of networked storage are
`rooted in technology that has existed for several decades, 40
`there are recent developments in computing which have lead
`to drastically increased demands for storage space. This is
`due in a large part to the advent of massive unstructured data
`flows, observed to stem from the mass acceptance of the
`internet and its related methods of communications and 45
`dissemination of information, as opposed to those associated
`with structured data flows, typical examples of which are
`Enterprise Resource Planning (ERP) systems, and Elec(cid:173)
`tronic Document Interchange (EDI) systems. There is a
`growing need to effectively control and manage data storage so
`activities related to the former, the latter being more easily
`able to be controlled and managed. As such, new challenges
`are brought forth related to facilitating server and storage
`consolidation, non-disruptive back-up procedures, and mini(cid:173)
`mizing the distance limitations of technologies preceding
`storage area networking.
`This need has led to the development of what is com(cid:173)
`monly referred to as a 'Storage Area Network' (SAN).
`Within a SAN, host computers provide access to arrays of
`storage devices that can be either local or remotely located, 60
`and can be either centralized in one location or distributed
`over many. This variability and the complexity of such
`storage subsystems mandates that the host computers be
`coupled to devices that can route requests to the storage
`devices and make their actual configuration transparent to
`the end-user (e.g., 'storage routers'). This added network
`(the SAN) between the hosts and the storage devices, when
`
`SUMMARY OF THE INVENTION
`
`In one embodiment of the present invention, a distributed
`storage management platform (DSMP) architecture is dis(cid:173)
`closed. Such a DSMP architecture includes a number of
`storage routers. Each one of the storage routers comprises a
`ss number of interface controllers. One of the interface con-
`trollers of each one of the storage routers is communica(cid:173)
`tively coupled to one of the interface controllers of at least
`one other of the storage routers.
`The foregoing is a summary and thus contains, by
`necessity, simplifications, generalizations and omissions of
`detail; consequently, those skilled in the art will appreciate
`that the summary is illustrative only and is not intended to
`be in any way limiting. As will also be apparent to one of
`skill in the art, the operations disclosed herein may be
`65 implemented in a number of ways, and such changes and
`modifications may be made without departing from this
`invention and its broader aspects. Other aspects, inventive
`
`Petitioners Microsoft Corporation and HP Inc. - Ex. 1040, p. 20
`
`

`

`US 6,877,044 B2
`
`3
`features, and advantages of the present invention, as defined
`solely by the claims, will become apparent in the non(cid:173)
`limiting detailed description set forth below.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`5
`
`4
`channel controller invention (per MPFCC patent
`application). This embodiment is a developed construct
`implementation of the most basic open construct of DSMP
`Architecture shown in FIG. 3. Each such device, described
`in earlier figures as a subsystem interface device, now
`becomes a storage router type device (or storage router).
`Each storage router connects one host to a storage array/
`subsystem, and these storage router devices are shown
`coupled in two separate loops, one each on the host side and
`10 the device side. The hosts are connected into the former
`loop, each through a fibre channel interface controller
`(mounted internally to each), whilst the storage array/
`subsystem elements are connected into the latter loop, each
`through a non-specific but nevertheless compatible interface
`15 means. From the perspective of the storage router device,
`each storage router is connected to each one of these two
`loops separately, through a host side multi-port fibre channel
`controller, and a device side multi-port fibre channel con(cid:173)
`troller respectively.
`FIG. 6 is a block diagram illustrating an example of
`connectivity which is an extension of that shown in FIG. 5.
`Additional to the elements described there, the items equiva(cid:173)
`lent to those subsystem interface devices as per FIG. 4, are
`now described as SVE devices, and are each shown to be
`25 connected to a sub-network, also to which the hosts are
`connected. Each element is linked to the network through a
`network interface card of a type specific to that element
`family. This is yet a further implementation of the invention
`which capitalizes on the ability that DSMP Architecture
`30 provides for inter-device communication across any of sepa(cid:173)
`rate independent channels. Thus, this embodiment of the
`invention, which is a complete construct implementation of
`DSMP Architecture ion applied to a SAN, enables an extra
`level of management flexibility and redundancy beyond the
`35 one shown in FIG. 5.
`FIG. 7 is a block diagram illustrating what constitutes an
`SVE device, a daughtercard and a motherboard, the first
`embracing a collective of components that make up a dual
`port fibre channel controller, the second encompassing an
`40 identical group of components but also accompanied by
`various others fundamental to the processing operations that
`must take place to permit the DSMPA to function.
`FIG. 8 is a schematic diagram which demonstrates how a
`DSMP Architecture may be employed in a SAN which is
`45 constructed using other components in addition to storage
`router devices and their links, such as switches and hubs.
`FIG. 9 is a schematic diagram illustrating a comparison
`between three different types of architecture which can be
`employed in a SAN, in such a way that their contrasting
`50 aspects are emphasized. The first two represent existing art,
`the third shows the DSMP Architecture.
`The use of the same reference symbol in different draw(cid:173)
`ings indicates similar or identical items. The use of the same
`label suffixes (i.e., digits beyond the first one that that
`55 coincides with FIG. #) in different drawings also indicates
`similar or identical items.
`
`20
`
`The present invention may be better understood, and its
`numerous objects, features, and advantages made apparent
`to those skilled in the art by referencing the accompanying
`drawings.
`FIG. 1 is a schematic diagram illustrating the basic
`philosophy underlying a SAN employing a DSMP Archi(cid:173)
`tecture. The SAN is shown by a 'cloud' symbol (a common
`representation of the potentially complex coupling of links),
`which can for example incorporate storage networking
`devices that may exist in a storage area network. The
`implication is that the details of connectivity therein can be
`temporarily overlooked while examining the hosts and stor(cid:173)
`age drives array /subsystems attached separately external to
`the SAN. Within the cloud is a collage of subsystem
`interface devices, each containing two different interface
`controllers, without s