I 1111111111111111 11111 111111111111111 11111 11111 1111111111 lll111111111111111
`
`United States Patent
`[19J
`
`US006118776A
`
`Berman
`
`[11]Patent Number:
`6,118,776
`[45]Date of Patent:
`Sep.12,2000
`
`[54]METHODS AND APPARATUS FOR FIBER
`
`CHANNEL INTERCONNECTION OF
`PRIVATE LOOP DEVICES
`
`Primary Examiner-Huy D. Vu
`
`
`
`Assistant Examiner---M. Phan
`
`Attorney, Agent, or Firm-Lyon & Lyon LLP
`
`
`
`
`
`
`[75]Inventor:
`Calif.
`
`Stuart B. Berman, Newport Beach,
`[57]
`
`ABSTRACT
`
`
`[73]Assignee:
`
`
`[21] Appl. No.: 08/907,385
`[22]Filed:Aug. 7, 1997
`
`
`
`
`
`Related U.S. Application Data
`
`Methods and apparatus for Fiber Channel interconnection is
`
`
`
`
`Vixel Corporation, Bothell, Wash.
`
`
`
`
`
`provided between a plurality of private loop devices through
`
`
`
`
`a Fiber Channel private loop device interconnect system. In
`
`
`
`
`the preferred embodiments, the Fiber Channel private loop
`
`
`device interconnect system is a fabric or an intelligent
`
`
`bridging hub. In one aspect of this invention, a Fiber
`
`[ 63]Continuation-in-part of application No. 08/801,471, Feb. 18,
`
`
`
`
`Channel private loop device is connected to two or more
`
`
`
`
`Arbitrated Loops containing, or adapted to contain, one or
`H04L 12/56; [51]Int. Cl.7 ........................... H04L 12/28;
`
`
`
`
`
`more private loop devices. Preferably, the interconnect sys­
`
`H04L 12/54; G06F 13/42
`
`
`tem includes a routing filter to filter incoming Arbitrated
`
`.......................... 370/351; 370/401; 370/422; [52]U.S. Cl.
`
`
`
`
`
`Loop physical addresses ( ALPAs) to determine which Fiber
`
`
`370/425; 370/351; 395/200; 395/286
`
`
`
`
`
`Channel frames must attempt to be routed through the fabric.
`[58]Field of Search .....................................
`
`370/401, 403,
`
`370/404, 405, 406, 422, 424, 425, 407,
`
`
`
`
`Numerous topologies of interconnect systems may be
`
`
`351, 352, 355; 395/200, 286; 359/161,
`
`
`
`
`achieved. In another aspect of this invention, a method is
`197, 200
`
`
`
`
`provided for implementing a logical loop of private loop
`
`
`
`
`devices by segmenting the logical loop into a plurality of
`
`
`
`sets, assigning each set to a physical Arbitrated Loop and
`
`
`
`
`connecting the Arbitrated Loops to a Fiber Channel private
`4,821,034 4/1989 Anderson et al. .
`
`
`
`
`
`
`loop device interconnect system. Additional methods are
`
`
`4,958,341 9/1990 Hemmady et al.
`370/60.1
`....................
`
`5,412,653 5/1995 Hoppe et al. .
`
`
`
`
`provided for restricting attached devices to Arbitrated Loop
`
`5,418,780 5/1995 Henrion.
`
`
`
`
`
`physical addresses within certain ranges. Additionally,
`
`
`
`5,432,907 7/1995 Picazo, Jr. et al. ..................... 359/161
`
`
`
`
`methods are provided for resetting hosts, the method gen­
`
`
`5,490,007 2/1996 Bennett et al. .
`
`5,502,719 3/1996 Grant et al. .
`
`
`
`
`erally comprising the steps of detecting the addition of a
`5,519,695 5/1996 Purohit et al. .
`
`
`
`storage device to a first Arbitrated Loop, and thereafter,
`
`
`
`
`resetting the Arbitrated Loop or loops on which a host or
`
`
`
`hosts reside on second Arbitrated Loop. Methods for opera­
`
`
`
`tion with use of SCSI initiators generate a link service reject
`
`
`when no address match is found, or when an address match
`
`
`
`is found, but where no device with the destination ALPA
`
`
`
`exists on the Arbitrated Loop corresponding to the destina­
`
`
`
`Malavalli, Kumar, "Distributed Computing With Fibre
`tion.
`
`
`
`
`Channel Fabric", Digest of Papers, IEEE Compean, Feb.,
`1992.
`
`
`
`
`1997.
`
`[56]
`
`
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(List continued on next page.)
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`0 709 987 A2 5/1996 European Pat. Off . .
`
`
`
`
`
`
`0 876 075 A2 11/1998 European Pat. Off . .
`
`OTHER PUBLICATIONS
`
`
`
`(List continued on next page.)
`
`
`
`81 Claims, 16 Drawing Sheets
`
`FIBRE CHANNEL FABRIC BLOCK DIAGRAM
`
`
`
`468 BRIDGED NETWORK
`
`456 FIBER OR COPPER PORTS
`
`4 77 FIBER OR COPPER PORTS
`
`478 FIBER OR COPPER PORTS
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 1 of 32
`
`

`

`6,118,776
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`OIBER PUBLICATIONS
`
`C.R. Martin, "Fabric Interconnection of Fibre Channel Stan­
`
`
`
`
`dard Nodes", Proceedings of the SPIE, vol. 1784, Sep. 8,
`
`
`5,528,584 6/1996 Grant et al. .
`199, pp 65-71.
`
`5,535,035 7 /1996 DeFoster et al. . ... ... ... ... .... ... ... 359 /161
`
`
`
`5,598,541 1/1997 Malladi ................................... 395/286
`G. Ravindran et al., "A Comparison of Blocking and Non­
`
`
`
`
`
`
`
`5,619,497 4/1997 Gallagher et al. ...................... 370/394
`
`
`
`5,638,518 6/1997 Malladi ................................... 395/200
`
`
`
`Blocking Packet Switching Techniques in Hierarchial Ring
`
`
`
`5,751,715 5/1998 Chan et al. ............................. 370/455
`
`Networks", IEICE T ransactions on Information and Systems
`
`
`
`5,768,530 6/1998 Sandorfi .................................. 395/200
`
`1, vol. E79-D, No. 8, Aug., 1996, pp 1130---1138.
`
`
`
`
`5,841,990 11/1998 Picazo, Jr. et al. ..................... 395/200
`
`
`5,894,481 4/1999 Book ....................................... 370/412
`
`
`Malavalli, K., "High Speed Fibre Channel Switching Fabric
`
`
`Services", Proceedings of the SPIE, vol. 1577 Sep. 4, 1991,
`"Fibre Channel Executive Overveiw", issued by Fibre Chan­
`
`
`
`pp. 216-225.
`nel Association.
`PCT International Search Report, dated Jul. 20, 1998, con­
`
`
`"Fibre Channel Overview", by Zoltan Meggyesi of Research
`
`
`
`
`
`Institute for Particle and Nuclear Physics.
`
`
`
`sisting of a total of four ( 4) sheets.
`
`
`
`G.R. Stephens et al., "Fibre Channel. The Basics", Jun.
`Platt et al., T raffic Management In Frame Relay Networks,
`
`
`
`1995, Ancot Corporation, USA, pp 5-6 to 5-9.
`
`
`
`
`Computer Networks and ISDN Systems, vol. 23, No. 4, Jan.
`
`
`W.Richard, "Fibre Channel as a Network Backbone",
`
`Wescon, Sep. 27, 1994, pp 659-659.
`1, 1992, pp. 305-316.
`
`
`H.Chin, "Fibre Channel Offers Another Road to High­
`Stephens et al., "Fibre Channel. The Basics" Jun. 1995,
`
`
`
`
`
`
`Speed Networking and 1/0", Computer Technology Review,
`Ancot Corporation, USA, pp 5-6 to 5-9.
`
`
`vol. 15, No. Suppl, Dec. 21, 1995, pp 45-47.
`
`
`A.Varma et al., "Using Camp-On to Improve Te Perfor­
`Fibre Channel Fabric Loop Attachment (FC-FLA) Rev 2.7,
`
`
`
`
`
`mance of a Fibre Channel Switch", Proceedings of the
`
`
`
`NCITS working draft proposed Technical Report, Aug. 12,
`
`
`
`Conference on Local Computer Networks, Jan. 1, 1993, pp
`1997.
`247-255.
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 2 of 32
`
`

`

`JBOD 9
`
`d •
`
`r:JJ. • � � ...... �
`
`= ......
`
`HOST HOST
`
`41 42
`[Q] [Q]
`I □I I □
`
`'JJ.�
`
`10
`1-+I I +-+I l-+-1 1-+I I +-+I I
`JBOD 3
`1l
`JBOD 6
`HOST HOST HOST
`I I I 11111 I 7 111111111
`....,........,. 4
`12
`111111111
`23 24 25
`111111111
`[Q]OO
`111111111
`8
`13
`111111111
`I I I I I I I I I
`I I I I I I I I I
`{
`I
`14
`I c::p 9 � 9? L.-J c::p LJ c:p c:JC::::i I AGS/8 2
`
`AGS/8 1
`
`o B
`�
`
`RAID
`19
`
`I □I
`I
`HOST
`HOST
`17
`18
`
`HOST
`
`20
`
`&.1
`
`'?
`'"""'
`
`
`�N
`
`N
`0
`0
`0
`
`'JJ.
`=­� �
`
`....
`
`'"""'
`
`0 ....,
`
`'"""'
`O'I
`
`0--,....
`
`....
`.....:a
`.....:a
`0--,
`
`AGS/8 40
`46
`
`45
`
`o B
`
`o B
`
`IH�11S11S 1I�
`�
`�
`[Q] [Q] [Q]
`-----511 □I
`JBOD RAID
`43 44
`AGS/8 37
`I 4 ry � =f � er �
`CJ I
`&.2
` � � 00
`
`22
`
`21
`
`o B
`
`26
`
`27 28 RAID
`29
`HOST HOST HOST
`
`30 31 32
`
`I □I' L...JII □
`HOST HOST HOST
`33 34 35 RAID
`36
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 3 of 32
`
`

`

`6,118,776
`U.S. Patent Sep.12,2000
`
`Sheet 2 of 16
`
`LOOP
`NODE 55
`
`LOOP
`NODE 56
`
`LOOP
`NODE 57
`
`LOOP
`NODE 58
`
`51
`
`60
`
`52
`
`61
`
`53
`
`62
`
`54
`
`59
`
`PLOGI FOR
`ALPA 83
`HOST
`
`JBOD
`HUB JBOD
`
`□
`
`□□□□□□□□
`
`78 70 82 71
`
`72 73
`
`PORT
`1
`
`PORT 2 PORT 3 PORT 4
`
`74 FABRIC
`
`ROUTER 76
`SWITCH CORE 77
`
`81
`
`79
`
`INTERNAL FABRIC
`
`
`CONTROLLER 75
`
`LS_RJT RESPONSE 80
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 4 of 32
`
`

`

`6,118,776
`U.S. Patent Sep.12,2000
`
`Sheet 3 of 16
`
`HOST
`
`HOST
`
`91
`
`92
`
`HOST
`□ □ I
`104)
`95
`
`I
`I
`
`I
`
`I'------"
`
`□ I I
`PLOGI FOR
`
`99 98
`
`ALPA 96
`
`103
`
`97
`
`FABRIC
`
`90
`
`i
`102 INTERNAL FABRIC
`
`ROUTER
`101 CONTROLLER
`SWITCH CORE
`LS_RJT RESPONSE
`
`93
`94
`
`100
`
`95
`
`JBOD
`
`125
`
`HOST HOST HOST
`1-+-+-+-t-+++-I ..............................
`131
`122 123 124
`[Q] [Q] [Q] 132
`
`.-==-. □ .-==-. □ .-==-. □
`
`133
`
`126
`
`127
`
`128
`
`129
`
`130
`
`1-1-1-1-1-1-1-1-1
`
`L...L...L...L.L..L..L...L..J
`
`D
`
`HOST
`
`i..=-=-�'=-1=,=-=:+--==-:::s;:=-----=
`
`�� AGS/8
`c::::J =--=
`
`c::::J
`
`=....i
`
`120
`
`121
`
`□I I□I
`I 0 1 I
`HOST HOST HOST
`
`136 137 138
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 5 of 32
`
`

`

`d •
`r:JJ. •
`
`�
`�
`......
`
`= ......
`
`'JJ.
`�
`'?
`'"""'
`
`
`�N
`
`N
`0
`0
`0
`
`'JJ.
`=­�
`� ....
`,i;;..
`0 ....,
`
`'"""'
`O'I
`
`0--,....
`
`
`�
`�
`00
`....
`.....:a
`.....:a
`0--,
`
`JBOD 146
`HOST HOST HOST
`152
`143 144 145 14 7 H-++-IH-+-+-1
`HOST HOST HOST HOST
`
`153
`148 �
`162 163 164 165
`[Q] [Q] [Q]
`149
`154
`1 □ 1 □ I □ 1
`155
`I 150 � �
`[Q] [Q] [Q] [Q]
`- - - - - - - - "
`151
`156
`�� 1 AGS/8 141
`[Q]
`D �� 1AGS/8 1401 � g � � � g �
`� D □rr �
`D HOST
`142
`
`\.
`
`I □I I □I I D
`
`1-1-t-t-,t-+++-I
`
`t-t-t-+-+-t-t-+-t
`
`111111111
`I
`
`I HOST HOST
`157 158
`£ii.. 7
`
`I 0 11 °1 I 1
`
`HOST HOST HOST
`159 160 161
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 6 of 32
`
`

`

`d•
`r:JJ.•
`�
`�
`...... �
`
`= ......
`
`'JJ.
`�
`'?
`
`'"""'
`
`
`�N
`
`N
`0
`0
`0
`
`'JJ.
`=­�
`�
`....
`Ul
`
`0
`....,
`'"""'
`O'I
`
`�.8
`
`
`
`Cl AGS/8 202
`
`JBOD 208
`209
`--210
`1-++He++H
`211
`1-++He++H
`1-++++++H 212
`213
`
`...-----I
`
`AGS/8 200
`1 _ =r =r ::-r T T _ _ _ _
`1
`
`<-=-=-==-=='-=-=-=T=F---==-=::...JI
`
`
`
`AGS/8 203
`
`.----------'
`, _______ __L __ ,
`
`
`
`AGS/8 20
`
`
`
`AGS/8 206
`
`AGS/8 207
`
`DDDDDDD E3E3
`--
`DD
`
`HOST HOST HOST
`
`HOST HOST HOST
`
`HOST HOST HOST
`
`==
`
`e_ __
`0--,....
`HOST HOST HOST
`�
`�
`00....
`.....:a
`.....:a
`0--,
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 7 of 32
`
`

`

`6,118,776
`U.S. Patent
`
`Sheet 6 of 16
`Sep.12,2000
`
`JBOD 221
`227
`22 8 i-+++-++++,
`
`232
`229
`RAID 222
`230 § 231
`
`AGS/8 220
`□□□□□□ BB
`DD
`CJD
`c:::::J c:::::J
`c:::::J c:::::J
`
`AGS/8 241
`□□□□□□ BB
`DD
`CJ CJ
`c:::::J c:::::J
`c:::::J c:::::J
`
`□
`
`'----------lL_....!::
`
`'------------1
`
`�---------1
`
`233 □ 237
`D:!..t-------'
`HOST 223
`234 □ 238
`DI----------'
`HOST 224
`235 □ 239
`HOST 225
`236 □ 240
`HOST 226
`
`- -
`
`---'
`
`01----
`
`D
`
`JBOD 261 RAID 262
`267_- B
`268
`B
`
`□
`
`1....1..J,..LLJ,.J...J....J
`
`272
`□�� AGS/8 260
`
`c:::::J=-= c:::::J =--.J
`
`L...::::1=--=-=-=:c::..-=:i::...==---=--=1=----=
`
`�.11
`I
`I D II D II._____,
`HOST
`HOST HOST HOST
`264 265 266
`263
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 8 of 32
`
`

`

`d •
`
`r:JJ. • � �
`...... �
`
`= ......
`
`'JJ.�
`
`'?
`'"""'
`
`
`�N
`
`N
`0
`0
`0
`
`'JJ.
`=­��
`
`....
`-..J
`
`0
`
`....,
`'"""'
`O'I
`
` � � 00
`
`0--,....
`
`....
`.....:a
`.....:a
`0--,
`
`HOST HOST HOST
`
`HOST HOST HOST
`
`HOST HOST HOST
`
`HOST HOST HOST
`
`AGS/8 250
`
`AGS/8 255
`
`AGS/8 256
`
`AGS/8 257
`
`AGS/8 258
`□□□□□DD BB
`
`□□
`
`==
`
`□
`
`BHOST HOST
`
`RAID
`
`� HOST HOST
`
`RAID
`
`BHOST HOST
`
`RAID
`
`BHOST HOST
`
`RAID
`
`Cl
`
`h.10
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 9 of 32
`
`

`

`d •
`r:JJ. •
`
`�
`�
`...... �
`
`= ......
`
`'JJ.
`
`� '?
`
`'"""' �N
`
`
`
`N
`0
`0
`0
`
`'JJ.
`=­�
`�
`....
`00
`
`0
`
`....,
`'"""'
`O'I
`
`0--,....
`00 ....
`
`��
`
`
`
`.....:a.....:a
`
`
`0--,
`
`RAID � HOST HOST
`
`RAID � HOST HOST
`
`,19,J9,
`,19,A
`
`RAID
`� HOST HOST
`AA
`
`RAID
`� HOST HOST
`AA
`
`AGS/8 300
`
`T =r I AGS/8 301 , � + � +
`
`
`
`AGS/8 306
`
`AGS/8 307
`
`AGS/8 308
`
`AGS/8 309
`E3 E3□□
`
`= ==
`
`Cl
`
`B HOST HOST
`RAID
`
`c
`
`B HOST HOST
`RAID
`
`a
`
`B HOST HOST
`RAID
`
`Cl
`
`B HOST HOST
`RAID
`
`h.12
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 10 of 32
`
`

`

`6,118,776
`U.S. Patent Sep.12,2000
`
`Sheet 9 of 16
`
`WORKSTATION
`DISK
`□
`ARRAY
`
`PC
`
`□
`
`□
`:: WORKSTATION
`
`□
`FABRIC 402
`
`PC
`
`□ □ □
`
`DISK ARRAYS 404
`
`
`
`MAINFRAME 405
`
`FABRIC CONTROL
`
`TO BRIDGE MODULE 455
`
`488 ROUTER &
`DUART 484 SRAM 482 DRAM 483
`�-PORT CONTROL
`.....---�-----�-�----
`489 MODULES
`FLASH 485 PROCESSOR
`481
`
`----
`
`DECODE/DMA
`CONTROL -----490
`MODULE 487
`
`&.15
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 11 of 32
`
`

`

`d •
`r:JJ.•
`
`�
`�
`...... �
`
`= ......
`
`'JJ.
`�
`'?
`'"""'
`
`
`�N
`
`N
`0
`
`8
`
`'JJ.
`=­�
`�
`....
`'"""'
`0
`
`0
`
`....,
`'"""'
`O'I
`
`0--,....
`
`
`�
`�
`00
`....
`.....:a
`.....:a
`0--,
`
`FIBRE CHANNEL FABRIC BLOCK DIAGRAM
`
`468 BRIDGED
`NETWORK
`l--
`
`-
`
`I
`
`�.JJ
`
`MODULE 455
`BRIDGE
`I
`476 471
`!
`I FABRIC
`CONTROL
`454
`INTERNAL
`465
`t 464
`PORT
`466 i ROUTER 452
`CONTROL
`470
`I
`467
`!463 459 t �460
`469
`457 PORT CONTROL
`456 FIBER OR COPPER PORTS
`451
`,__ -
`458
`TO PORT CONTROL 465
`461 t ! 462 t
`SWITCH
`PORT CONTROL
`4 77 FIBER OR COPPER PORTS
`CORE
`474 ,__ i---
`453
`
`,__ -
`
`� i---
`
`I
`I
`I
`I
`I
`I
`I
`
`TO PORT CONTROL 465
`472 t ! 473 t
`PORT CONTROL
`478 FIBER OR COPPER PORTS
`--475 --
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 12 of 32
`
`

`

`PROCESSOR 1/F
`
`TO SWITCH CORE PROCESSOR 1/F
`
`1522
`523 - � 543
`
`ADDRESS TABLE
`542 STATISTICS
`r--.i ROUTER CONTROL
`532
`(RCFSM) 540
`(RSG) 541
`545 I TO INTERNAL ROUTER
`1509
`�.JUJ
`
`ADDRESS MATCH
`
`(ADM) 531
`
`MODULES 547
`PORT CONTROL
`rF.J.�
`PORT CONTROL �
`ROUTE REQUEST 506 505
`ROUTE
`
`502 ROUTE REQUEST 1/F (ROUTE REQUEST)
`510 SELECTOR 508
`'"""'�N
`DETERMINATION
`ROUTE REQUEST (PCRRIM) 530 501
`(RDM) 538 SELECTED (RRS) 537
`N000
`507
`ROUTE .....,__ ___ ____,J
`
`51215131 REQUEST UNBLOCKED ROUTE
`REQUEST 515 ,---- �-- ---
`, . t
`BLOCKED ROUTE REQUEST
`REQUESTING
`ROUTE STATE
`TABLE (RST) _ ROUTE REQUEST 516 TABLE (BRTBL)
`PORT
`533
`539 524 UNBLOCK
`517
`DETERMINATION
`511 I I (RRUNB) BLOCKED ROUTE REQUEST
`ARRAY H 503
`536 518 PORT REGISTER
`
`(BRRA) 534
`
`PROCESSOR I/F
`521
`I BLOCKED ROUTE REQUEST� 504
`1
`
`
`TIMER (BRTMR) 535 SELECTED PORT
`519
`FROM RRS 546
`f
`I/F 548
`PROCESSOR
`PORT CONTROL �14
`
`ROUTE RESPONSE 1/F PORT CONTROL
`
`(PCRSPM) 544 (ROUTE RESPONSE)
`
`rF.J.
`=­� � ....
`'"""''"""'
`0 ...., '"""'
`
`d
`r:JJ.
`
`•
`
`•
`
`�
`
`� ...... �
`=......
`
`O'I
`
`0--,
`.... �
`�
`00
`.....:a
`.....:a
`0--,
`
`....
`
`FABRIC ROUTER
`
`1
`
`I
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 13 of 32
`
`

`

`PORT CONTROL
`PROCESSOR 1/F
`
`PROCESSOR 1/F
`
`.L·.1..1.·v
`FIFO
`' ROUTER 1/F
`OVERRUN
`570
`569
`564
`563 t !
`PREVENTION
`LOGIC
`PORT 559
`561
`FIF0555 I 554 I TAG
`CONTROL
`TO SWITCH
`MODULE 560
`562
`I
`t I
`I
`CORE
`556
`ENDEC
`558 571 DATA
`553
`
`>--
`
`OPTICS OR
`COPPER
`GLM552
`
`GIGABIT 1/F
`
`FROM SWI
`CORE
`
`565
`566
`
`PROCESSOR/DATA 557 567
`
`
`ARBITRATION LOGIC (PDAL)
`1
`
`PROCESSOR 1/F
`
`17
`
`LIFA FRAME FORMAT
`
`
`
`SOFiL I 22 00 00 EF I 00 00 00 EF I O 1 38 00 00 I 00 00 00 00 I FF FF FF FF I 00 00 00 00
`
`�-- 11 02 00 00 I 16 BYTE (128 BIT) ALPA MAP I CRC I EOFt
`600
`
`h.18
`
`d•
`r:JJ. •
`�
`� ......
`
`......
`
`'JJ.
`�
`
`'"""' �N
`N 0
`
`'JJ.
`=­�
`
`� .... '"""' N
`0 ...., '"""' O'I
`
`0--,....
`�
`�
`00....
`
`.....:a
`.....:a
`0--,
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 14 of 32
`
`

`

`d •
`r:JJ. •
`�
`�
`EXAMPLE ALLOWING ONLY THE ALPA'S 10, 17, 18, 1B, 1D, 1E AND 1F TO BE CHOSEN
`......�
`= ......
`
`
`
`
`
`
`
`SOFiL I 22 00 00 EF 00 00 00 EF 01 38 00 00 00 00 00 00 FF FF FF FF 00 00 00 0
`
`LIFA FRAME FORMAT
`
`
`
`.___ ____ 11 02 00 00 I 7E 03 FF FF FF FF FF FF FF FF FF FF I CRC I EOFt
`
`16 BYTE (3 WORD) ALPA MAP 602
`
`601
`
`�.19
`
`USM FRAME FORMAT
`
`'JJ.�
`'? '"""'�N
`N0
`8
`
`'JJ.
`=­� �
`
`....'"""'� 0
`
`EXAMPLE WORLD WIDE PORT NAME OF ALL ZEROS TO OBTAIN LOOP INITIALIZATION MASTERSHIP
`
` ...., '"""'
`
`
`
`SOFiL I 22 00 00 EF I 00 00 00 EF 01 38 00 OP 00 00 00 Ok> FF FF FF FlF 00 00 00 0
`
`O'I
`
`0--,....
`
` �
`�
`00
`....
`.....:a
`.....:a
`0--,
`
`8 BYTE WORLD WIDE PORT NAME
`
`�----11 01 oo oo I oo oo oo oo oo oo oo o EOFt
`
`
`610
`
`�.20
`
`
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 15 of 32
`
`

`

`6,118,776
`U.S. Patent Sep.12,2000 Sheet 14 of
`16
`
`PORT 2 PORT 4 PORT 6 PORT 8
`
`642 644 646 648
`
`AGS/8 640
`
`c:::J c:::J
`c:::J c:::J
`
`□□
`c:::J c:::J
`
`641 643 645 647
`
`PORT 1 PORT 3 PORT 5 PORT 7
`
`£ii.. 21
`
`ARBITRATED LOOP
`
`DEVICE 700
`
`ARBITRATED LOOP
`
`DEVICE 701
`
`_,-
`------
`�,�
`',,,
`706
`705
`--
`✓'
`',
`'
`
`\
`\
`\
`\
`\
`I
`I
`
`,I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`
`\
`
`FABRIC OR
`LOGICAL I
`INTELLIGENT
`ARBITRATED
`BRIDGING
`LOOP I
`HUB 704
`709 \
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`\
`\
`\
`\
`
`'
`708
`., .,
`707
`',,
`,,.,
`, ___ ......
`
`ARBITRATED LOOP
`DEVICE 703
`
`---------
`
`ARBITRATED LOOP
`
`DEVICE 702
`
`£ii.. 2J
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 16 of 32
`
`

`

`6,118,776
`U.S. Patent
`
`Sep.12,2000
`
`Sheet 15 of 16
`
`ARBITRATED
`ARBITRATED
`ARBITRATED
`ARBITRATED
`LOOP
`LOOP
`LOOP
`LOOP
`DEVICE
`DEVICE
`DEVICE
`DEVICE
`672
`673
`670
`671
`
`676
`I
`I PORT BYPASS I PORT BYPASS I PORT BYPASS I PORT BYPASS I
`1
`1
`CIRCUIT I CIRCUIT I CIRCUIT I CIRCUIT
`L ________ _j__ ________ _j__ ________ _j__ ________ �
`
`HUB 678
`
`679
`
`h.22
`
`{PRIOR ART)
`
`FIBRE
`FIBRE
`FIBRE
`CHANNEL
`CHANNEL
`
`DISK
`DISK
`DISK
`681
`682
`680
`
`FIBRE
`CHANNEL CHANNEL
`DISK
`683
`
`686
`I
`I PORT BYPASS I PORT BYPASS I PORT BYPASS I PORT BYPASS I
`1
`I
`I
`I
`1
`
`CIRCUIT CIRCUIT
`CIRCUIT
`CIRCUIT
`L ________ _j__ ________ _j__ ________ _j__ ________ �
`
`JBOD 688
`
`689 690
`
`CABINET INTERFACE
`691
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 17 of 32
`
`

`

`d
`r:JJ.
`
`•
`
`•
`
`�
`�
`......
`
`......
`
`'JJ.
`�
`
`'"""'
`
`
`�N
`
`N
`0
`
`'JJ.
`=­�
`�
`....
`'"""'
`O'I
`
`0
`
`....,
`'"""'
`O'I
`
`0--,
`.... �
`�
`00
`
`....
`.....:a
`.....:a
`0--,
`
`765
`
`766
`
`767
`
`768
`
`ARBITRATED ARBITRATED ARBITRATED
`ARBITRATED
`LOOP DEVICE LOOP DEVICE LOOP DEVICE LOOP DEVICE
`
`---------- ---------1....--------
`-----
`r
`------7
`I
`I
`,.._ PORT BYPASS PORT BYPASS PORT BYPASS PORT BYPASS I
`I
`I
`I
`CIRCUIT 753
`CIRCUIT
`
`CIRCUIT 751 CIRCUIT 752
`754
`I
`I
`_ _j
`L--------------------------------------
`
`
`
`HUB SUBMODULE 774
`
`
`
`ALPA FILTERING
`PORT 760
`
`-
`
`763
`PROCESSOR PRIMITIVE ROUTING INTELLIGENT
`
`
`BRIDGING HUB
`CONTROL 762 FUNCTION 761
`773
`
`--
`
`764
`
`-A
`
`LPA FILTERING
`PORT 759
`
`r----------.---------------------------
`-7
`I
`I
`PORT BYPASS PORT BYPASS PORT BYPASS PORT BYPASS
`� I
`I --
`756 CIRCUIT 757
`
`
`CIRCUIT 758 I
`I
`
`CIRCUIT 755 CIRCUIT
`I
`I
`L--------------- -----------------______
`_J
`
`
`
`HUB SUBMODULE 775
`
`ARBITRATED ARBITRATED ARBITRATED
`ARBITRATED
`LOOP DEVICE LOOP DEVICE LOOP DEVICE LOOP DEVICE
`
`769
`
`770
`
`771
`
`772
`
`h.25
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 18 of 32
`
`

`

`1
`
`
`
`METHODS AND APPARATUS FOR FIBER
`
`6,118,776
`
`
`
`CHANNEL INTERCONNECTION OF
`
`PRIVATE LOOP DEVICES
`
`FIELD OF THE INVENTION
`
`
`
`
`
`2
`loop devices attempt a Fabric Login (FLOGI) upon initial­
`
`
`
`
`
`
`
`
`ization. Public loop devices also are cognizant of all twenty
`
`
`
`
`
`four bits of the 24-bit NL_Port native port address identifier.
`
`
`
`Public loop devices will open the fabric port at Arbitrated
`
`
`
`
`5 Loop Physical Address ( ALPA, bits 7 to 0) zero when the
`RELATED APPLICATION INFORMATION
`
`
`domain and area (bits 23 to 8) do not match their domain and
`
`
`This application is a continuation-in-part of Application
`
`
`
`area. Private loop devices use only the lower eight bits of the
`
`
`
`Ser. No. 08/801,471 filed Feb. 18, 1997, entitled "Fibre
`
`
`
`ALPA and can only communicate within the local loop.
`
`
`
`
`Channel Switching Fabric", now pending, which is incor­
`
`
`
`Generally, the disadvantages of the Arbitrated Loop topol-
`
`
`
`
`porated herein by reference as if fully set forth herein.
`
`
`
`
`10 ogy include: first, it is a blocking topology, that is, only a
`
`
`
`
`
`single connection between a pair of nodes is allowed at any
`
`
`point in time (excluding the broadcast mode). Second,
`
`
`
`
`
`The present invention relates to input/output channel and
`
`
`
`
`
`
`device buffering occurs in each device as it has a six word
`
`
`
`networking systems, and more particularly to methods of
`
`
`
`
`buffer, creating a delay of up to 225 nanoseconds. This delay
`
`
`
`
`using a Fibre Channel fabric or intelligent bridging hub to
`
`
`15 is additive with each device in the loop. The delay creates
`
`
`
`
`interconnect Fibre Channel Arbitrated Loops composed of
`
`
`
`overhead for the communicating devices when a large
`
`
`
`
`
`private loop devices (i.e., devices which do not support
`
`
`
`number of devices are connected to a loop. Third, distance
`
`direct fabric attachment).
`
`
`also adds delay to a loop and is additive for each device. For
`
`copper medium there is a 4 nanosecond delay per meter and
`BACKGROUND OF THE INVENTION
`
`
`
`20 for optical medium there is a 5 nanosecond delay per meter.
`
`
`
`Fourth, robustness is an issue since all devices are on one
`
`
`
`
`Fibre Channel is an American National Standards Institute
`
`
`
`(ANSI) set of standards which describes a high performance
`
`
`
`loop any device failure will cause the entire loop to fail or
`
`
`
`
`
`reset. Fifth, the total bandwidth available is limited to the
`
`serial transmission protocol which supports higher level
`
`storage and network protocols such as HIPPI, S CSI, IP, ATM
`
`
`
`
`
`
`
`
`bandwidth of the loop itself. Finally, device failure is an
`
`
`
`
`25 issue since while frames are being transmitted, a timeout in
`and others. Fibre Channel was created to merge the advan­
`
`
`
`
`
`
`an upper level protocol may occur, thereby disrupting the
`
`tages of channel technology with network technology to
`
`
`applications.
`create a new 1/0 interface which meets the requirements of
`
`
`
`both channel and network users. Channel technology is
`
`
`
`
`Loop devices are typically interconnected on an Arbi­
`
`
`
`
`
`
`usually implemented by 1/0 systems in a closed, structured
`
`
`
`trated Loop with a hub, see FIG. 22 numeral 678. The hub
`30
`
`
`
`
`and predictable environment, whereas network technology
`
`
`
`is a passive device, that is a loop exists within the hub 674,
`
`
`usually refers to an open, unstructured and unpredictable
`
`
`
`675, 676,677,679. A hub in most cases maintains the loop's
`
`
`
`
`integrity when devices are removed, powered off, or fail by
`environment.
`
`
`using a port bypass circuit 674, 675, 676, 677. Hubs simply
`
`
`
`Advantages of Fibre Channel typically include the fol­
`
`
`
`
`receive and redrive the signals to individual devices.
`
`
`
`
`lowing. First, it achieves high performance, which is a
`35
`
`
`
`critical in opening the bandwidth limitations of current
`
`There are many disadvantages which result when inter-
`
`
`
`
`
`computer to storage and computer to computer interfaces at
`
`
`
`
`connecting private loop devices with hubs: First, hubs do not
`
`
`
`speeds up to 1 gigabit per second or faster. Second, utilizing
`
`
`
`
`
`address the blocking nature of the loop topology. Second,
`
`
`
`
`
`jitter is propagated from bypassed nodes. This additive affect
`
`
`
`fiber optic technology, Fibre Channel can overcome tradi­
`
`
`
`
`tional 1/0 channel distance limitations and interconnect
`
`
`
`arewhen a large number of devices 40 causes loop instability
`
`
`
`
`devices over distances of 6 miles at gigabit speeds. Third, it
`
`
`
`interconnected. Third, when data is currently being trans­
`
`
`
`
`is high level protocol independent, enabling Fibre Channel
`
`
`
`
`ferred and a device attached to a hub is powered off or fails,
`
`
`
`
`to transport a wide variety of protocols over the same media.
`
`
`
`the loop could be reset which is destructive to the commu­
`
`
`
`Fourth, Fibre Channel uses fiber optic technology which has
`
`
`
`
`nicating devices. Fourth, if a device is inserted into a live
`
`
`
`a very low noise properties. Finally, cabling is simple in that
`
`
`loop the loop will be reset which is destructive to the
`45
`
`
`
`
`Fibre Channel typically replaces bulky copper cables with
`
`communicating devices.
`
`small lightweight fiber optic cables.
`The majority of initial Fibre Channel equipment deploy­
`
`
`
`
`
`
`Fibre Channel supports three different topologies: point­
`
`
`
`ment utilizes the Arbitrated Loop topology with hubs as the
`
`
`
`
`to-point, Arbitrated Loop and fabric attached. The point-to­
`
`
`
`interconnect. These environments are experiencing all the
`
`
`
`point topology attaches two devices directly. The Arbitrated
`
`
`
`inherent in both Arbitrated 50 previously defined problems
`
`
`
`Loop topology attaches devices in a loop. The fabric
`
`
`Loop topology and with hub deployment. The blocking
`
`
`
`
`
`attached topology attaches a device directly to a fabric.
`
`
`
`
`nature of the Arbitrated Loop is limiting the number of
`
`
`
`
`devices on a loop. The distance and delay parameters are
`
`
`
`
`The Arbitrated Loop topology was initially designed to
`
`
`
`
`also creating more overhead for the loop. Finally the loop is
`provide a lower cost interconnect than fabrics and to provide
`
`
`55 being reset by single devices.
`
`
`more interconnect than point-to-point topologies. The Arbi­
`
`
`
`As such, it is the goal of this invention to provide
`
`
`
`
`trated Loop topology was created by separating the transmit
`
`and receive fibers associated with each loop port and con­
`
`
`
`
`apparatus and methods which solve or mitigate these prob­
`
`
`
`necting the transmit output of one loop port to the receive
`lems.
`
`
`
`input of the next loop port. Typically, characteristics of the
`SUMMARY OF THE INVENTION
`
`
`
`
`Arbitrated Loop topology include: first, it allows up to 126 60
`
`
`
`
`participating node ports and one participating fabric port to
`This invention relates to methods and apparatus for Fibre
`
`
`
`
`
`
`
`
`communicate, second, each node port implements a route
`
`
`
`
`Channel interconnection of a plurality of private loop
`
`
`
`filtering algorithm, and third, all ports on a single loop have
`
`
`
`
`devices through a Fibre Channel private loop device inter­
`
`
`
`the same upper 16 bits of the 24-bit NL_Port address
`
`
`
`
`connect system. In the preferred embodiments, the Fibre
`identifier.
`
`
`
`65 Channel private loop device interconnect system is a fabric
`
`
`
`
`or an intelligent bridging hub. Through these methods and
`
`
`
`There are two classifications of devices on an Arbitrated
`
`
`
`
`apparatus, multiple Fibre Channel Arbitrated Loops, the
`
`Loop: private loop devices and public loop devices. Public
`
`
`
`Google Exhibit 1009
`Google v. LS Cloud Storage Technologies
`IPR2023-00120, Page 19 of 32
`
`

`

`6,118,776
`
`4
`3
`loops containing one or more private loop devices, may be
`
`
`
`serves to route frames between one or more private loop
`
`
`
`
`
`
`
`interconnected even though on separate Arbitrated Loops.
`
`
`
`devices on a first Arbitrated Loop and one or more private
`
`
`loop devices on at least a second Arbitrated Loop.
`
`
`
`In the preferred embodiment, an interconnection system is
`
`
`
`Preferably, the method includes the steps of receiving the
`
`
`
`
`provided for connecting a plurality of physically separate
`
`
`
`at a 5 Fibre Channel frames over the first Arbitrated Loop
`
`
`
`Fibre Channel Arbitrated Loops, the loops either containing,
`
`
`
`
`connected port of a Fibre Channel private loop device
`
`
`or being adapted to contain, one or more private loop
`
`
`
`interconnect system and filtering the frame by, either, for­
`
`
`
`
`devices. The apparatus preferably includes at least a first
`
`
`
`warding the frame on the first Arbitrated Loop if the frame
`
`
`
`Arbitrated Loop containing, or adapted to contain, one or
`
`
`more private loop devices, and at least a second Arbitrated
`
`
`
`has an address on the first Arbitrated Loop, or, providing an
`
`
`
`
`
`Loop, either containing, or adapted to contain, one or more
`
`
`
`Loop if the address10 "open" response on the first Arbitrated
`
`
`
`private loop devices. The Arbitrated Loops are intercon­
`
`
`
`is not on the first Arbitrated Loop. Optionally, in the event
`
`
`
`
`nected via a Fibre Channel private loop device interconnect
`
`
`
`
`that the frame includes an address not on the first Arbitrated
`
`
`system which is disposed between the Arbitrated Loops.
`
`Loop, the additional step of attempting to route the frame
`
`
`
`
`
`In one embodiment, a Fibre Channel fabric is disposed
`
`
`
`
`through the Fibre Channel private loop device interconnect
`
`between the Arbitrated Loops, and includes a routing filter
`
`
`
`
`
`15 system may be made. In yet another optional step, buffering
`
`
`
`
`which filters incoming Arbitrated Loop physical addresses
`
`
`
`
`of the frames destined to private loop devices not on the first
`
`
`(ALPAs) to determine which Fibre Channel frames must
`
`
`
`Arbitrated Loop may be performed, most preferably, per­
`
`attempt to be routed through the fabric. Thus, by disposing
`
`
`
`mitting cut-through if the route can be made without sub-
`
`
`
`
`routing information regarding private loop devices within
`
`stantial buffering.
`
`
`
`
`
`the Fibre Channel private loop device interconnect system,
`Yet another novel method of these inventions is a method
`
`
`
`20
`
`
`
`
`multiple Arbitrated Loops containing private loop devices
`
`
`
`
`for restricting attached devices to Arbitrated Loop physical
`may be interconnected.
`
`
`
`addresses (ALPAs) within certain ranges. Through this
`
`
`
`Any type of private loop device, consistent with the
`
`
`
`
`method, multiple Fibre Channel Arbitrated Loops of private
`
`apparatus and methods stated herein, may be utilized in
`
`
`
`
`
`
`
`
`loop devices are configured, each private loop device on the
`conjunction with this system. Examples of private loop
`
`
`
`
`
`
`Arbitrated Loop physical 25 Arbitrated Loop having an
`
`
`
`
`devices include storage devices, such as tape drives, JBODs
`
`
`
`address. Generally, the steps in the preferred method
`
`
`and RAID subsystems, host systems, and other connections
`
`
`comprise, first, dividing the ALPAs into nonoverlapping
`
`
`
`
`within a system, such as bridges, particularly SCSI to Fibre
`
`
`sets, second, assigning each set to a separate physical
`
`
`
`
`Channel bridges, routers, particularly Fibre Channel to asy­
`
`
`
`Arbitrated Loop, and thereafter, during loop initialization,
`
`
`
`
`
`chronous transfer mode systems and Fibre Channel to eth­
`
`
`
`
`30 forcing the attached private loop devices to choose from the
`ernet systems.
`
`assigned set.
`
`
`
`
`Various interconnection topologies may be utilized with
`Yet another novel method comprises a method for reset­
`
`
`
`
`
`
`
`these systems. Beyond a single fabric having two Arbitrated
`
`
`
`
`ting hosts within a Fibre Channel interconnection system of
`
`Loops, any number of Arbitrated Loops may be utilized,
`
`
`
`
`private loop devices. In this method of operation of an
`
`
`
`consistent with the size constraints of the fabric. Alternately,
`
`
`
`more than one 35 interconnection system, the system including
`
`
`
`
`a first fabric, with at least one Arbitrated Loop attached, and
`
`
`
`
`Arbitrated Loop, at least one loop being adapted to contain
`
`
`a second fabric, with at least one Arbitrated Loop, may have
`
`
`
`
`
`storage and one loop adapted to contain a host, those devices
`
`
`
`the first fabric and second fabric directly connected.
`
`
`
`being private loop devices, the loops being connected to a
`
`
`
`
`
`Alternatively, or additionally, a first fabric and a third fabric,
`
`
`
`
`
`Fibre Channel private loop device interconnect system, the
`
`
`each having at least one Arbitrated Loop, may be connected
`
`
`
`
`40 method generally comprises the steps of, first, detecting at
`
`
`
`
`through a second fabric. Yet another topology includes a first
`
`
`least the addition of a storage device to a first Arbitrated
`
`
`
`
`
`fabric having M ports, where one port is connected to
`
`
`
`Loop, and thereafter, resetting the Arbitrated Loop or loops
`
`
`
`storage, preferably JBODs, and the remaining M-1 ports of
`
`
`
`
`on which a host or hosts reside upon such detected addition.
`
`
`
`the first fabric are connected to M-1 second fabrics. An
`
`
`
`In this manner, a host resident on an Arbitrated Loop
`
`
`
`
`alternative interconnect topology includes a first fabric with
`
`
`
`
`which have private loop devices 45 becomes aware of storage
`
`
`
`M ports, and M second fabrics, each second fabric being
`
`
`been added to other Arbitrated Loops separated from the
`
`
`
`connected to the first fabric. In yet another interconnect
`
`
`
`
`host bearing Arbitrated Loop by a Fibre Channel private
`
`
`
`
`
`topology, a device is connected to a first fabric by a first path
`
`loop device interconnect system.
`
`
`
`and to a second fabric by a second, independent path. In yet
`
`
`
`
`In yet another method of operation of the inventive
`
`
`
`
`another interconnect topology, a first set of fabrics and a
`
`
`50 system, a method for operation with use of SCSI initiators
`
`
`second set of fabrics may be interconnected through a first
`
`
`is provided. In this interconnection system, the system
`
`
`
`
`intermediate fabric and a second intermediate fabric, each of
`
`
`includes more than one Ar