United States Patent [19J
`Fu
`
`[54] APPARATUS AND METHOD FOR A CACHE
`COHERENT SHARED MEMORY
`MULTIPROCESSING SYSTEM
`
`[75]
`
`Inventor: Daniel D. Fu, Sunnyvale, Calif.
`
`[73] Assignee: HotRail, Inc., San Jose, Calif.
`
`[21] Appl. No.: 08/986,430
`
`[22]
`
`Filed:
`
`Dec. 7, 1997
`
`[51]
`[52]
`
`[58]
`
`[56]
`
`Int. Cl? ...................................................... G06F 12/08
`U.S. Cl. .............................. 710/100; 710/29; 710/30;
`710/52; 709/233; 711!110; 711!131; 711!149
`Field of Search ................................ 710/52, 100, 29,
`710/30; 709/233; 711!110, 131, 149
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,315,308
`4,438,494
`4,480,307
`5,313,609
`5,335,335
`5,440,698
`5,505,686
`5,511,226
`5,513,335
`5,524,234
`5,535,363
`5,537,569
`5,537,575
`5,553,310
`5,561,779
`5,568,620
`5,574,868
`5,577,204
`5,581,729
`5,588,131
`5,594,886
`5,606,686
`5,634,043
`5,634,068
`5,644,754
`5,655,100
`5,657,472
`5,682,516
`
`2/1982 Jackson . ... ... ... .... ... ... ... ... ... .... .. 364/200
`3/1984 Budde et a!.
`........................... 364/200
`10/1984 Budde eta!. .
`5/1994 Baylor et a!. ........................... 395/425
`8/1994 Jackson et a!. ......................... 395/425
`8/1995 Sindhu et a!.
`..................... 395/200.08
`4/1996 Tarui et a!.
`............................. 395/448
`4/1996 Zilka .. ... ... ... ... .... ... ... ... ... ... .... .. 395/823
`4/1996 McClure ................................. 395/457
`6/1996 Martinez, Jr. et a!. ................. 395/468
`7/1996 Prince ..................................... 395/474
`7/1996 Masubuchi .............................. 395/448
`7/1996 Foley eta!. ............................. 395/468
`9/1996 Taylor et a!.
`........................... 395/860
`10/1996 Jackson et a!. ......................... 395/449
`10/1996 Sarangdhar et a!. . ... ... ... .... ... ... 395/285
`11/1996 Marisetty . ... ... .... ... ... ... ... ... .... .. 395/298
`11/1996 Brewer et a!. ..................... 395/200.01
`12/1996 Nishtala et a!.
`........................ 395/470
`12/1996 Borrill ..................................... 395/473
`1!1997 Smith eta!. ............................ 395/463
`2/1997 Tarui et a!.
`............................. 395/448
`5/1997 Self et a!.
`............................... 713/503
`5/1997 Nishtala et a!.
`............................ 712/1
`7/1997 Weber eta!. ........................... 395!500
`8/1997 Ebrahim eta!. ........................ 395/471
`8/1997 VanLoo eta!. ........................ 395/485
`10/1997 Sarangdhar et a!. .................... 395/473
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US006065077 A
`[11] Patent Number:
`[45] Date of Patent:
`
`6,065,077
`May 16,2000
`
`5,684,977
`5,696,910
`5,796,605
`5,829,034
`5,895,495
`5,897,656
`5,940,856
`5,946,709
`
`11/1997 VanLoo eta!. ........................ 395/470
`12/1997 Pawlowski .............................. 395/280
`8/1998 Hagersten ............................... 711!143
`10/1998 Hagersten eta!. ...................... 711!141
`4/1999 Arimilli et a!. ......................... 711!156
`4/1999 Vogt eta!. .............................. 711!141
`8/1999 Arimilli et a!. ......................... 711!119
`8/1999 Arimilli et a!. ......................... 711!119
`
`OTHER PUBLICATIONS
`
`Technical White Paper, Sun™ Enterprise™ 10000 Server,
`Sun Microsystems, Sep. 1998.
`Alan Charlesworth, Starfire: Extending the SMP Envelope,
`IEEE Micro, Jan./Feb. 1998, pp. 39-49.
`Joseph Heinrich, Origin™ and Onyx2™ Theory of Opera(cid:173)
`tions Manual, Document No. 007-3439-002, Silicon
`Graphics, Inc., 1997.
`White Paper, Sequent's NUMA-Q SMP Architecture,
`Sequent, 1997.
`White Paper, Eight-way Multiprocessing, Hewlett-Packard,
`Nov. 1997.
`
`(List continued on next page.)
`
`Primary Examiner-Ayaz R. Sheikh
`Assistant Examiner--Eric S. Thlang
`Attorney, Agent, or Firm-Bennett Smith
`
`[57]
`
`ABSTRACT
`
`The system and method for operating a cache-coherent
`shared-memory multiprocessing system is disclosed. The
`system includes a number of devices including processors,
`a main memory, and 1!0 devices. Each device is connected
`by means of a dedicated point-to-point connection or chan(cid:173)
`nel to a flow control unit (FCU). The FCU controls the
`exchange of data between each device in the system by
`providing a communication path between two devices con(cid:173)
`nected to the FCU. The FCU includes a snoop signal path for
`processing transactions affecting cacheable memory and a
`network of signal paths that are used to transfer data between
`devices. Each signal path can operate concurrently thereby
`providing the system with the capability of processing
`multiple data transactions simultaneously.
`
`55 Claims, 160 Drawing Sheets
`
`200
`~
`
`212
`
`(FCU)
`
`NETAPP, INC. EXHIBIT 1015
`Page 1 of 191
`
`

`
`6,065,077
`Page 2
`
`01HER PUBLICATIONS
`
`George White & Pete Vogt, Profusion, a Buffered, Cache(cid:173)
`-Coherent Crossbar Switch, presented at Hot Interconnects
`Symposium V, Aug. 1997.
`Alan Charlesworth, et al., Gigaplane-XB: Extending the
`Ultra Enterprise Family, presented at Hot Interconnects
`Symposium V, Aug. 1997.
`James Loudon & Daniel Lenoski, The SGI Origin: A
`ccNUMA Highly Scalable Server, Silcon Graphics, Inc.,
`presented at the Proc. of the 24th Int'l Symp. Computer
`Architecture, Jun. 1997.
`Mike Galles, Spider: A High-Speed Network Interconnect,
`IEEE Micro, Jan./Feb. 1997, pp. 34-39.
`T.D. Lovett, R.M. Clapp and R.J. Safranek, NUMA-Q: An
`SCI-based Enterprise Server, Sequent, 1996.
`Daniel E. Lenoski & Wolf-Dietrich Weber, Scalable
`Shared-Memory Multiprocessing, Morgan Kaufmann Pub(cid:173)
`lishers, 1995, pp. 143-159.
`
`David B. Gustavson, The Scalable Coherent Interface and
`Related Standards Projects, (as reprinted in Advanced Mul(cid:173)
`timicroprocessor Bus Architectures, Janusz Zalewski, IEEE
`Computer Society Press, 1995, pp. 195-207.).
`Kevin Normoyle, et al., UltraSPARC™ Port Architecture,
`Sun Microsystems, Inc., presented at Hot Interconnects III,
`Aug. 1995.
`Kevin Normoyle, et al., UltraSPARC™ Port Architecture,
`Sun Microsystems, Inc., presented at Hot Interconnects III,
`Aug. 1995, UltraSparc Interfaces.
`Kai Hwang, Advanced Computer Architecture: Parallelism,
`Scalability, Programmability, McGraw-Hill, 1993, pp.
`355-357.
`Jim Handy, The Cache Memory Book, Academic Press,
`1993, pp. 161-169.
`AngelL. DeCegama, Parallel Processing Architectures and
`VLSI Hardware, val. 1, Prentice-Hall, 1989, pp. 341-344.
`
`NETAPP, INC. EXHIBIT 1015
`Page 2 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 1 of 160
`
`6,065,077
`
`..
`...
`
`/
`co
`0
`-r-
`
`..
`...
`
`/
`<0
`0
`-r-
`
`...
`~·
`
`/
`c
`C\1
`0
`-r-
`a:s
`C\1
`0
`-r-
`\..
`..
`...
`
`E~
`CD CJ
`....... 0
`en_
`~()
`(j)
`
`/
`
`0 F
`
`0
`-r-
`
`~
`
`0
`........
`
`....._
`....
`
`__..
`
`~
`
`~
`c
`0
`a:s E
`~ CD
`~
`
`....._
`.....
`
`__..
`
`~
`
`~~~c
`-r- o
`en
`a:s
`CD
`u CJ
`e a..
`
`....
`.....
`
`__..
`
`~
`
`1:::
`<(
`s.....
`0 ·c
`c..
`.
`C> ·-u.
`
`•
`•
`•
`
`-r-
`
`-r- o
`a:s
`
`en ~
`....
`CD
`
`~liD~
`() CJ e a..
`
`'\
`
`..
`
`~
`
`.....
`
`NETAPP, INC. EXHIBIT 1015
`Page 3 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`N
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`2148
`
`214A
`
`..
`[
`
`•
`
`216B
`
`.. c
`
`Mcuo •
`
`c2oaA c216A •I
`
`I•
`
`MCUL I /16N-~~ Memory I
`/2;6N •I Memory I
`Me~ory I
`Me~ory I
`
`\:214N-1
`
`•
`
`\: 214N
`
`·
`
`•
`
`L.._ __
`
`.
`.
`.
`
`210N+L-1
`
`\
`
`..
`
`I BBUK ~206K
`
`Fig. 2
`
`,..
`...
`
`I
`
`21BK./
`
`....
`
`1 BBUo ~20~~ •
`
`,..
`...
`
`218A7
`
`I
`
`.....
`....
`
`210N+L
`
`\
`
`210N+L+2
`
`210N+L+K-1
`
`..
`
`\
`210N+1 I
`
`~
`
`~
`
`""" 210N+2
`
`..
`~
`208N
`
`\
`
`Processoro r'-202A I Processorn ~202N
`
`• • •
`
`(FCU)
`
`Flow Control Unit
`
`212
`I
`
`~8A
`
`I
`
`ctuo
`
`210~
`
`~
`200
`
`NETAPP, INC. EXHIBIT 1015
`Page 4 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`~
`
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`'
`
`I
`
`Fig. 3
`
`Memory Writeback
`
`CIU -> FCU -> MCU
`
`Non-cacheable read or write
`
`BBU -> FCU -> MCU
`
`Read Invalidate
`
`110 read or write
`
`CIU-> FCU
`
`CIU -> FCU -> BBU
`
`CIU -> FCU -> BBU
`
`110 memory read or write
`
`BBU-> FCU-> MCU (cache miss)
`
`cacheable write
`
`Cacheable read
`
`or
`
`BBU -> FCU (cache hit)
`
`CIU -> FCU -> MCU (cache miss)
`
`CIU -> FCU (cache hit)
`
`Transaction
`
`Initiator-> Target
`
`NETAPP, INC. EXHIBIT 1015
`Page 5 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 4 of 160
`
`6,065,077
`
`cache read
`or write miss
`
`cache hit or
`write data
`
`cache miss/
`write data
`
`Fig. 4A
`
`NETAPP, INC. EXHIBIT 1015
`Page 6 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 5 of 160
`
`6,065,077
`
`BBU
`
`CIU
`
`1/0 memory
`read or write
`
`cache hit or
`write data
`
`FCU
`
`read
`data
`
`cache miss/
`write data
`
`MCU
`
`Fig. 48
`
`NETAPP, INC. EXHIBIT 1015
`Page 7 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 6 of 160
`
`6,065,077
`
`1/0 memory
`read/write
`
`1/0 memory
`read/write
`
`read
`data
`
`read
`data
`
`CIU
`
`FCU
`
`BBU
`
`Fig. 4C
`
`NETAPP, INC. EXHIBIT 1015
`Page 8 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 7 of 160
`
`6,065,077
`
`CIU
`
`1/0 read/write
`
`read data
`
`FCU
`
`1/0 read/write
`
`read data
`
`BBU
`
`Fig. 40
`
`NETAPP, INC. EXHIBIT 1015
`Page 9 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 8 of 160
`
`6,065,077
`
`• • •
`
`CIUn
`
`cache hit
`
`FCU
`
`cache miss
`
`MCU
`
`Fig. 4E
`
`NETAPP, INC. EXHIBIT 1015
`Page 10 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 9 of 160
`
`6,065,077
`
`CIU
`
`memory
`
`write back
`
`FCU
`
`write
`data
`
`Fig. 4F
`
`non-cacheable
`read or write
`
`FCU
`
`non-cacheable
`,___ __ read or write
`
`MCU
`
`Fig. 4G
`
`NETAPP, INC. EXHIBIT 1015
`Page 11 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 10 of 160
`
`6,065,077
`
`220'\
`
`CPU
`
`J ..
`
`, r
`
`Processor1 222 '\
`L2 Cache
`Memory
`
`J ..
`
`, r
`
`r'\202
`
`224'\
`
`J
`
`I
`
`.h
`
`226'\ ~r
`
`228 '\
`
`A~
`
`, ,
`
`L2 Tag
`Memory
`
`L2 Cache
`Controller
`
`208'\
`
`.. ~
`
`230,
`~r
`Channel
`Control
`Unit
`
`232 '\
`
`CIU1
`
`.h.
`
`~,
`
`212 '\
`
`232'\
`
`Duplicate L2
`Tag memory
`
`CIU-IFU
`
`FCU
`
`Fig. 5A
`
`NETAPP, INC. EXHIBIT 1015
`Page 12 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 11 of 160
`
`6,065,077
`
`Processor1
`
`222
`\.
`
`L2 Cache
`Memory
`~ .. ,,
`... L2 Cache
`.... Controller
`
`220,
`
`CPU
`
`....
`.....
`
`.4~
`
`230,
`
`~,
`
`Channel
`Control
`Unit
`
`.4~ 2108
`.,,
`~
`
`CIU1
`.4~ 210A
`v
`.,,
`
`'2o 2
`
`1\208
`
`/"226
`
`L2 Tag Memory
`
`1\
`212
`
`232./
`
`CIU-IFU
`
`FCU
`
`Fig. 58
`
`NETAPP, INC. EXHIBIT 1015
`Page 13 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`'"""' N
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`FCU
`
`CIU-IFU
`
`Fig. 5C
`
`Tag memory
`Duplicate L2
`/232
`
`212'1 232
`
`Processor1
`
`Memory
`L2 Cache
`
`... ..
`
`222
`
`/
`
`t /230
`
`.,~..
`
`Control
`Channel
`
`Unit
`
`1.. ....
`
`.A ..
`
`Cache Bus
`I++ Backend
`/248
`
`11240
`
`CPU
`
`Controller
`L2 Cache
`
`t /228
`
`Memory
`L2Tag
`
`l/226
`
`Memory
`L1Tag
`
`Memory
`L 1 Cache
`
`1/246
`
`t/244
`
`--..
`~242
`
`CPU Core
`
`NETAPP, INC. EXHIBIT 1015
`Page 14 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 13 of 160
`
`6,065,077
`
`~210 ,206
`
`Channel
`Control
`Unit2
`
`{
`
`256
`.I
`
`,2548
`
`/
`'\.
`BBU Control Unit
`'\.
`Bus
`Control
`Units
`
`252A
`.1
`. . .
`
`Channel
`Control
`Unit1
`
`252
`
`. . .
`
`/
`
`254A,
`Bus
`Control
`Unit1
`
`1\218
`
`/252
`
`256
`
`/
`
`254
`v
`
`206,
`
`Fig. 6A
`
`210
`
`/
`
`Channel Control Unit
`l
`BBU Control Unit
`t
`Bus Control Unit
`
`'218
`Fig. 68
`
`NETAPP, INC. EXHIBIT 1015
`Page 15 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`'"""' ~
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`------------------~ -
`L --
`SIGNALS
`I
`I CONTROL
`2608
`I ADDR~SS
`I
`IT
`-.
`I DATA •
`I
`r-2708 I
`l
`I = = = = = = = = = ------J
`
`Fig. 7
`
`MCU 204
`
`""
`-,. DRAM CONTROL L..
`•
`2668
`
`1
`
`UNIT
`
`,...
`
`'
`t..
`
`1
`
`CONNECTION I~ ~ 1/0 UNIT
`
`UNIT,
`
`ECC
`
`--,.
`•
`
`READ BUFFER 1
`
`/ 2628
`
`')Ill'"'
`
`WRITE BUFFER1
`
`·~
`
`2648--'
`
`UNIT 1
`
`I c2728
`
`I
`
`•
`
`1
`
`I
`I
`~ I
`• INTERFACE
`21 DB,\ II
`CHANNEL
`
`110 UNIT0
`
`I
`270Al
`
`UNIT0
`
`CONNECTION
`
`ECC
`
`268A
`
`~M-------27~--------
`
`QUEUE
`OUTPUT
`
`CHANNEL
`
`272A
`
`-
`
`1
`
`SIGNALS
`CONTROL
`
`~++----1
`
`~
`ADDRESS
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`UNIT 0
`
`DRAM CONTROL
`
`~ QUEUE ~ /2688
`
`OUTPUT
`
`lJ
`2768
`
`..,...
`
`2168
`
`,--t-
`
`I
`
`QUEUE
`
`264A
`
`UNIT 0
`
`INTERFACE
`
`263
`L _
`I
`
`COMMAND L
`
`NETAPP, INC. EXHIBIT 1015
`Page 16 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 15 of 160
`
`6,065,077
`
`1 202A
`
`1 202N
`
`Processor1
`
`Processorn
`....,_ ___ ....,. • • • 1 - - - - - - -1
`
`CPUn
`CPU1
`~..--_"T"'"""" _ _.."2oaA...._______,r---___.
`)oaN
`
`\210A
`
`210Nr
`
`Flow Control Unit
`
`(FCU)
`
`205,
`
`MCU/
`MCU-IF
`
`MCU/
`MCU-IF
`
`...._
`
`~
`
`""'
`....
`
`214A
`\_
`...
`
`2148
`\_
`...
`
`214L-1\.
`
`Memory
`
`Memory
`
`...
`
`Memory
`
`214L
`\.
`
`Memory
`
`210 ~~
`\_
`
`\... 205
`
`206K
`
`AGP
`
`r---...z...._-~
`
`1
`
`1
`r--.....Z.....-"""-,
`
`206A
`
`AGP
`
`BBUk
`
`, ...
`PCI 64/66
`PCI 64/66
`....
`PCI32/66
`
`...
`
`PCI32/66
`
`Fig. 8
`
`NETAPP, INC. EXHIBIT 1015
`Page 17 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`'"""' ~
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`BBU
`
`/206
`
`CIU
`
`. ..
`
`r-2828
`
`--...
`r-282A ...
`--...
`r-280A..,
`
`FCU CNTL2
`
`FCU CLK2
`
`212
`
`1280F
`r-282F
`..
`.... FCU_BBU DATA r-284~
`
`~
`
`I""
`_... BBU CLK
`I""
`.... BBU CNTL
`
`1
`
`-,.
`r-282E
`-,.
`r-280E
`
`FCU CNTL1
`
`FCU CLK1
`
`r-2808
`
`""''
`......_ CIU CLK
`""''
`l..oL CIU_CNTL
`~
`.... FCU CIU DATAr-284A
`
`MCU CNTL
`.... FCU MCU DATAr-284C FCU
`.... FCU CNTLO
`""''
`.... FCU CLKO
`
`MCU CLK
`
`~
`
`I""'~
`
`MCU
`
`r-204
`
`Fig. 9
`
`...
`r-280Q
`...
`r-282q_
`...
`
`r-282C
`
`r-280C
`
`NETAPP, INC. EXHIBIT 1015
`Page 18 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`'"""'
`-..J
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`Fig. 10
`
`RSVD
`
`Address[31 :3]
`
`l31l3ol29l2al27l2sl2sl24l23l22l21l2ol19l1al17l1sl1sl14l13l12l11l1 ol 91 al 11 sl sl41 31 211 I o I
`I IIIII III T 1111 TIll IIIII T r111 T llll
`
`l_ Ll _l ...L L l_j _i _L Ll_j
`
`Ll_j _l l_ Ll _l ...L L l_j ..l _L Ll_j i
`
`ls3l62ls1lso ls9lsals7lsslssls4ls3ls2ls1lso l49l4al4 7l4sl4sl44l43l42l41l4ol39l3al37l3sl3sl34l33l32l
`I IIIII III T 1111 TIll IIIII T r111 T llll
`
`...L L l_j _l _L L _jj_l_
`
`[4:0]
`
`DeviceiD
`_l l_ Ll
`
`D c L
`
`[35:32]
`Address
`l_l Ll_j_i Ll_j
`
`CMD[4:0]
`
`[2:0]
`QueueiD LEN
`
`[3:0]
`
`BE[7:0]
`
`NETAPP, INC. EXHIBIT 1015
`Page 19 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 18 of 160
`
`6,065,077
`
`~
`<(
`0
`
`0
`z
`<(
`~
`~
`0
`(.)
`
`T"""
`T"""
`
`m
`.
`C)
`u.
`
`<(
`T"""
`.
`T"""
`·-u.
`
`C)
`
`~
`<(
`0
`
`0
`z
`<(
`~
`~
`0
`(.)
`
`i..:
`0
`+""'
`co
`:;::;
`c:
`
`+""'
`Q)
`~
`co
`I-
`
`L..
`0
`+""'
`co
`;.....
`c:
`
`..;...;
`Q)
`~
`co
`I-
`
`NETAPP, INC. EXHIBIT 1015
`Page 20 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 19 of 160
`
`6,065,077
`
`~
`
`(.) z
`>(cid:173)en
`
`~
`
`~
`...J
`1-
`(.)
`1-
`
`~
`0
`...J
`1-
`(.)
`1-
`
`~
`
`(.) z
`>(cid:173)en
`
`<(
`C\1
`~ .
`·-LL
`
`C)
`
`~
`
`(.) z
`>(cid:173)en
`
`0 > en
`
`~
`
`~
`~
`...J
`1-
`(.)
`0::
`
`~ z
`>(cid:173)en
`
`m
`C\1
`~ .
`·-LL
`
`C)
`
`NETAPP, INC. EXHIBIT 1015
`Page 21 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`"""" ~ c
`0 ......,
`N c
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`"""" ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`204.1
`
`MCU2
`
`... ....
`
`... ....
`
`...
`....
`
`MCU1
`
`... ....
`204'\
`
`Fig. 13
`
`FCU 212
`
`'-318K+N
`
`'-318E
`
`'-3180
`
`~318C
`
`,~3188
`
`....
`....
`MCU_IFU4 -
`
`3120
`
`....
`....
`MCU_IFU3 -
`
`312C
`
`....
`....
`MCU_IF~ -
`
`3128
`
`....
`....
`MCU_IFU1 -
`
`312A
`
`350
`
`CLOCK
`SYSTEM
`
`l
`
`~ ....... ..
`~ ..... ..
`~ .....
`~ ..... .. ~
`·~ !
`310N
`I
`·~
`
`CIU_IFUn
`
`. ..
`
`88U_IFUk_1 88U_IFUk CIU_IFU1
`, .. I
`·~
`
`330K
`
`·~ !
`~,. I
`
`~
`
`. !
`I ~,.
`·~
`
`330K-1
`
`310A
`
`,
`'-318A
`~ ~ ~ ~ ~
`3648~ h h h ~ ~
`.... h ~ h ~ ~
`f1 ~· ~ k1 ~·
`! ·~
`I ,.
`
`3640~
`
`364C"./
`
`364A'./
`
`!
`...
`
`3308
`88U_IFU1 88U_IFU2
`I
`
`330A
`
`~
`
`CIUN
`
`CIU1
`
`BBUK
`
`208
`
`208
`
`206
`
`BBU1
`
`206
`
`NETAPP, INC. EXHIBIT 1015
`Page 22 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`'"""' 0 ......,
`
`N
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`...
`>
`....
`.)372
`
`,-
`
`336./
`
`334./
`
`Output Queue
`
`Command
`
`~lr
`
`Datal
`"'I 7
`
`Command
`
`Snoop
`
`Queue
`
`"'ii 7
`
`~ lr
`
`Fig. 14
`
`.,
`
`llo..
`
`...
`.J.
`
`332./
`
`Logic Unit
`Control
`Snoop
`
`0
`
`Lock buffer
`328\. Transaction
`{r
`
`+
`
`..(t
`Array
`
`")...
`
`Tag
`~ I
`
`Memory
`
`~DREQ#
`~SREQ#
`
`314./
`
`Control Unit
`
`Input
`Channel
`
`I
`
`...
`
`A
`
`Decoder
`Command
`
`"'
`
`A
`
`I
`
`323,
`
`210./
`
`~ Command
`Deferred
`324,
`
`Queue
`
`II
`
`PP-SMP Channel
`
`<
`
`.J.
`
`v37o
`
`flo.
`
`.J.
`
`...
`Data/Command A
`
`Input Queue
`
`v3ao
`
`326\.
`
`~ Decoder
`Command
`
`322,
`
`31{
`
`Control Unit
`
`Channel
`
`~ Output
`
`310
`
`NETAPP, INC. EXHIBIT 1015
`Page 23 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`N
`N
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`n +2
`
`21b,.
`
`>
`_....
`
`"'
`....
`
`Interface
`Channel
`
`Unit
`
`/_3508
`
`"'I; 7
`
`jl..
`
`.J.
`
`h. ,..
`...
`
`Fig. 15
`
`'-3408
`
`"'
`
`A
`
`Buffer
`
`Output Data
`
`Input Data
`
`Buffer
`
`,..
`
`h.
`
`/3428
`
`346./
`
`320 ~
`
`3648
`
`"'
`
`A
`
`l/312
`
`~348
`
`Control
`MCU IF
`
`Unit
`
`"'I; 7
`~ )t..
`
`..
`...
`
`..
`...
`
`l
`
`./
`
`210n +1
`
`',.
`
`h. ,>
`
`"'
`
`A
`
`Interface
`Channel
`
`Unit
`
`...
`,
`....
`
`/350A
`
`Decoder
`,.
`h. Command
`
`-
`
`'Ill
`
`A
`
`Queue
`Header
`
`344/ t I
`
`,..
`)
`_....
`
`342ft/
`
`Buffer
`
`Input Data
`
`,.
`
`h.
`
`340A
`
`"'
`I<
`Output Data ~
`
`Buffer
`
`A n n
`
`364A
`
`"'
`
`NETAPP, INC. EXHIBIT 1015
`Page 24 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`~
`N
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`336./
`Output Queue
`
`Command
`
`Datal
`
`..
`.. ")
`l/372
`
`I
`
`,.,
`
`'II 7'
`
`~DREQ#
`~SREQ#
`
`314
`
`Channel '
`
`Control Unit
`
`Input
`
`'II
`...
`
`Decoder
`Command
`
`'II
`
`A
`
`Command
`Deferred
`
`Queue
`
`~
`
`I
`
`323,
`
`324,
`
`210./
`
`11
`
`PP-SMP Channel
`
`Fig. 16
`
`334./
`
`" r
`
`... ~
`
`332./
`
`Queue
`..
`.. Command
`Snoop
`" ,.
`
`, ..
`
`....
`""'
`
`Logic Unit
`
`Control
`Snoop
`
`....; 7
`~ )
`
`....
`....
`
`Lock Buffer Array
`
`Transaction
`
`... r
`
`~"" ,,.
`
`.
`
`\
`328
`
`l/330
`
`-,
`
`<
`
`A
`
`lJ370
`
`;lo.
`
`...
`
`Input Queue
`
`Data/Command
`
`Decoder
`Command
`
`~
`
`322,
`
`316./
`
`Control Unit
`
`~ Output
`
`Channel
`
`300
`
`NETAPP, INC. EXHIBIT 1015
`Page 25 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 24 of 160
`
`6,065,077
`
`.. ~
`·-
`0
`~ ~
`~ (.) ... (.)
`....J
`....J
`I
`~
`~
`:::>
`....J
`....J
`....J
`(.)
`(.) (.)
`0 u.
`0
`
`~
`~
`
`~~
`
`(.) en
`
`~ ..
`
`~~
`
`I
`:::>
`(.)
`u.
`
`~
`(.)
`
`0 -(.)
`E
`Q)
`+J en
`>. en
`
`~~
`~
`(.)
`
`0 - ~
`(.) LO
`Q) M -
`(.) ca
`c: c:
`Q) C)
`s..... ·-
`~ en
`Q)
`0!:
`
`0
`LO
`M
`
`.
`C)
`u.
`
`.
`·-u.
`
`C)
`
`~ ~
`.....J
`....J
`(.)(.)
`00
`
`NETAPP, INC. EXHIBIT 1015
`Page 26 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`Ul
`N
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`----------------------~
`
`1
`
`Clock Domain 2
`
`'206K
`
`ssuk
`
`•••
`
`'206A
`
`ssu0
`
`L
`
`1
`
`Fig. 19
`
`I -~~ -~;------,;--~;---
`"' J
`= I
`
`-
`
`-
`
`.4~
`
`·--
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-1--
`
`-
`
`-
`
`.4~ Clock Domain 1 J ~
`
`.4~
`
`
`11
`
`...,.
`
`FCU
`
`I
`
`I
`
`"'
`
`~
`
`..:_
`
`,,
`
`,,
`
`II
`
`L
`I
` 212.-~
`I
`I ---------------l
`
`I
`I
`I
`I
`1
`I
`I
`/ 208A I
`I
`
`..
`: MCUI
`
`_...
`
`: r2oaL
`
`..
`~ MCUo
`~
`r
`
`I
`
`1
`I
`
`J
`
`I
`I
`
`I
`
`I
`
`I
`
`l
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`~
`
`200
`
`L _ ~ .. _______ J~ ____ l
`
`c1u0
`I --,,--
`L202A./
`"'202N 1
`_____________ ..:...J
`I
`I
`,-------------l
`
`'208N
`
`-
`
`-
`
`-
`
`n
`c1u
`-~,.. -
`
`~208A
`• • •
`
`-
`
`-
`
`-
`
`-
`
`1
`
`.4~
`
`Clock Domain 3
`
`J~
`
`Processorn
`
`• • •
`
`Processor0
`
`NETAPP, INC. EXHIBIT 1015
`Page 27 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`~
`N
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`MCU-IFU -
`• • •
`
`312L
`
`_ ... ...
`.....
`...
`
`~~
`
`312A ~
`
`MCU-IFU
`
`... .....
`
`...
`... .....
`
`.....
`
`.....
`.....
`.....
`....
`
`Fig. 20
`
`368
`
`.,,
`
`•••
`
`358
`
`-
`
`368
`-,r
`
`CONTROLLER
`
`SWITCH
`
`DATA PATH
`
`.....
`...
`
`.,,
`
`·u
`
`""
`....
`.....
`.....
`.....
`"" ...
`....
`""
`
`...ol
`
`360
`
`ARBITRATOR
`
`PATH
`SNOOP
`
`CLOCK 351
`SYSTEM
`
`-
`
`J ..
`
`~~
`
`310N
`
`CIU-IFU
`
`• ••
`
`J ..
`
`~ ..
`310A
`
`CIU-IFU
`
`~ ~
`300A
`
`~..
`
`BBU-IFU
`
`•••
`
`300A
`-+ BBU-IFU
`
`,r
`~ ..
`
`NETAPP, INC. EXHIBIT 1015
`Page 28 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' 0'1 c
`0 ......,
`-..J
`N
`~ .....
`'JJ. =-~
`
`N c c c
`
`'"""'
`~0'1
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`f Devicen+l+k
`
`Interface
`
`Device1
`Interface
`
`Fig. 21
`
`....
`... Destl D n+l+k [3:0
`]
`
`......-
`... DREQ n+l+k #
`
`•
`• •
`
`Controller
`
`Switch
`Path
`Data
`
`....
`... DestiD[3:0]
`
`....
`... DREQ#
`
`358
`
`....
`.... HPATH_IDm[1 :0]
`....
`.... NEm#
`....
`....
`
`NODE DIRm
`
`-
`
`Switch
`Node
`Destination
`To
`
`....
`... DGNTm#
`
`....
`... HPATH_IDn[1 :0]
`......
`.... NEn#
`....
`... NODE DIRn
`....
`... DGNTn#
`
`Switch
`Node
`Requestor
`To
`
`NETAPP, INC. EXHIBIT 1015
`Page 29 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 28 of 160
`
`6,065,077
`
`Interface data path
`318
`
`364A
`
`NEn#--.......-t
`
`NODE DIRn ""'-t'-"--.
`DPATH_IDn(O)
`DPATH_IDn(1) ...rt----'
`
`3648
`
`NEn#--.......-t
`
`NODE DIRn .......---.
`DPATH_IDn(O)
`DPATH_IDn(1) ---1---
`
`• • •
`
`3640
`
`NODE_DIRn '""'-t---.
`DPATH_IDn(O)
`DPATH_IDn(1)~--
`
`Fig. 22
`
`Node Switch
`368Ao
`
`Node Switch
`368A1
`
`Node Switch
`368A3
`
`NETAPP, INC. EXHIBIT 1015
`Page 30 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 29 of 160
`
`6,065,077
`
`0
`0
`
`C\1
`0
`
`,....
`0
`
`(1)
`0
`
`0
`(.)
`
`(\J
`(.)
`
`,....
`(.)
`
`(1)
`(.)
`
`.
`·-u..
`
`C)
`
`~
`...J
`(.)
`Cf)
`
`0
`:::>
`-
`(.)
`(§)
`~
`a
`w
`a:
`0
`
`0
`:::>
`-
`(.)
`(§)
`.....
`~
`z
`(!J
`0
`
`~
`...J
`(.)
`0
`
`<(
`1-
`<(
`0
`I
`0 :r:
`a..
`
`~
`~
`...J
`()
`0
`
`<(
`1-
`<(
`0
`I
`,....
`:r:
`a..
`
`NETAPP, INC. EXHIBIT 1015
`Page 31 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`~ c
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`Fig. 24
`
`364D
`364C
`3648
`364A
`
`369
`
`Flow Meter Unit
`
`'360
`
`361
`
`Data Path Interface Unit
`
`Arbitrator
`Snoop Path
`
`CPU
`
`358
`
`Controller
`
`Switch
`
`Data Path
`
`1367
`
`1
`
`Interface Umt
`Snoop Path.
`
`1
`
`I
`
`320
`
`NETAPP, INC. EXHIBIT 1015
`Page 32 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 31 of 160
`
`6,065,077
`
`Fig. 25A
`C_ SNOOP OPERATION 1
`
`~...
`
`...~
`
`II
`ADDRESS
`REQUEST PHASE) RESULT
`PHASE
`(, PHASE
`......
`...
`SA[63:0] ___ _____, f© ~ I
`SCLK I
`
`A1 7
`SNOOP _HIT[3:0] ______ _____,
`S(A1)L
`
`(
`
`il
`!t---
`:r---
`
`SBOFF#
`SCYCLE#
`
`SLOCK# - - - - - - - - - -+ (cid:173)
`
`SREQ1 # t®L--A __ -----:::::/r--------+-
`
`SGNT1# ---~L__B _____ f-
`REQUEST PHASE FOR SNOOP OPERATION2 J j
`
`SREQ2#
`
`...
`
`.
`
`SGNT2#
`I
`REQUEST PHASE FOR SNOOP OPERATION3 ~I
`_j!
`SREQ3#
`...
`I
`
`SGNT3#
`
`STARGET _ID - - - - - - - - - - -+ - -
`
`NETAPP, INC. EXHIBIT 1015
`Page 33 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 32 of 160
`
`6,065,077
`
`Fig. 258
`
`I
`
`I
`
`II
`I
`
`I
`
`I
`
`I
`j ~ SNOOP
`SNOOP
`i LOPERATION2
`t0PERATION3
`I
`l 'I ADDRESS
`II REQUEST
`: .. PHAS~ RESULT , .. PHAS2 RESULT
`PHASE
`PHASE
`!
`___..
`...
`...
`..
`...1
`,
`I
`I
`I
`I
`II,-----:__
`_j
`.
`I
`~ A2J
`~ E(A2) r;
`-i)\
`
`H
`:
`f\
`
`A3:7
`
`S(A3)~ : /.---
`:I
`I
`
`I
`
`~ 1\~------------~-----
`Devicel D ~---"'E""'-------._ __ ____ ----.-__ _
`
`NETAPP, INC. EXHIBIT 1015
`Page 34 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 33 of 160
`
`6,065,077
`
`Fig. 25C
`
`View A
`
`View B
`
`NETAPP, INC. EXHIBIT 1015
`Page 35 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`~
`~
`
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`ommand=
`
`invalidate
`
`read
`
`cacheable
`Command=
`
`read
`
`cacheable
`Command=
`
`write
`
`Clear lock
`
`buffer
`
`412
`
`Fig. 26
`
`Snoop Hit
`
`Backoff
`
`414
`
`410
`
`write back
`memory
`Command=
`
`read/write
`1/0 memory
`Command=
`
`462
`
`command
`
`Broadcast address/
`
`408
`
`r--
`
`Place address in lock buffer
`
`snoop bus
`
`Obtain access to
`
`1/0 memory read/write
`read/write, invalidate,
`Command = cacheable
`
`406
`
`404
`
`402
`
`CIU Interface Unit receives transaction and/or data from channel
`
`400
`
`NETAPP, INC. EXHIBIT 1015
`Page 36 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 35 of 160
`
`6,065,077
`
`Command=cacheable read
`
`.,,.
`
`Await deferred read reply
`
`.,
`Respond to deferred read
`reply and change state of
`cache line
`
`., r
`
`Clear lock buffer
`
`4 16
`
`1\
`418
`
`1\ 420
`
`1\
`422
`
`,,.
`Transmit read data to CIU ' 424
`.,,
`Return
`
`Fig. 27
`
`NETAPP, INC. EXHIBIT 1015
`Page 37 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 36 of 160
`
`6,065,077
`
`( Command=cacheable write
`,,
`
`,,
`
`h426
`
`,,
`
`Snoop hit=I/S/SWT/IWT
`
`Snoop hit=E/M/MC
`
`'-428
`
`'-436
`
`Obtain communication
`path to memory
`
`Retrieve target
`identifier
`
`'-430
`
`'-438
`
`Transmit write data
`
`'-432
`
`Obtain communication
`path to target device
`'-440
`
`+
`
`Clear lock buffer
`
`Target write data
`
`'-434
`
`.,,
`Return
`
`+
`
`'-442
`
`Clear lock buffer
`t
`Return
`
`'-444
`
`Fig. 28
`
`NETAPP, INC. EXHIBIT 1015
`Page 38 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 37 of 160
`
`6,065,077
`
`Command=read invalidate )\
`446
`,
`
`Await deferred read reply
`
`~,
`
`Respond to deferred read
`reply and change state of
`cache line
`
`~ ..
`
`Transmit read data to CIU
`
`.,,.
`
`Clear lock buffer
`
`' 450
`
`' ll-52
`
`' ll-54
`
`[\ 456
`
`~,
`
`Return
`
`Fig. 29
`
`NETAPP, INC. EXHIBIT 1015
`Page 39 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 38 of 160
`
`6,065,077
`
`~ Command=l/0 memory }.,_
`read/write
`462
`
`110 memory read
`
`1/0 memory write
`
`,,.
`
`Obtain communication
`path to target
`BBU- IFU
`
`Obtain communication
`path to target
`BBU -IFU
`
`'464
`
`, ..
`
`Transmit read transaction
`to BBU - IFU and await
`deferred read reply
`
`Transmit transaction
`and write data to
`BBU- IFU
`
`'4aa
`
`,lr
`Return
`
`Respond to deferred
`read reply and transmit
`read data to CIU
`,,
`Return
`
`Fig. 30
`
`NETAPP, INC. EXHIBIT 1015
`Page 40 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 39 of 160
`
`6,065,077
`
`(
`
`Command=
`non-cacheable
`read/write
`
`non-cacheable write
`
`,,.
`
`non-cacheable read
`
`,,
`Obtain communication
`path to MCU-IFU
`,,
`Transmit address
`command, and
`and write data
`,,
`Return
`
`'423
`
`,,
`
`Obtain communication
`path to MCU-IFU
`
`,,
`
`Transmit read address
`and command on
`communication path
`
`,,
`
`Await deferred
`read reply
`
`,,
`
`'433
`
`Respond to deferred read
`reply, obtain channel, and
`transmit read data to CIU
`,,.
`Return
`
`'435
`
`Fig. 31
`
`NETAPP, INC. EXHIBIT 1015
`Page 41 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 40 of 160
`
`6,065,077
`
`Command=
`1/0 read/write
`
`~482
`
`1/0 write
`1/0 read
`~r
`~--------~--------.
`
`Obtain communication
`path to target interface
`device
`
`Obtain communication
`path to target interface
`device
`
`'484
`
`Transmit
`address/command
`to device
`
`'485
`
`Await deferred
`read reply
`
`,,
`
`Transmit address/
`command/data to
`target interface device
`
`,,.
`Return
`
`Respond to deferred
`read reply and
`transmit data to CIU
`t
`Return
`
`'487
`
`Fig. 32
`
`NETAPP, INC. EXHIBIT 1015
`Page 42 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 41 of 160
`
`6,065,077
`
`Command=
`memory writeback
`
`~
`496
`
`'
`
`~,.
`
`Obtain communication
`path to memory
`
`1\ 498
`
`~,
`
`Place address in lock buffer 1\ 500
`
`~,
`
`Transmit write data
`
`,
`
`Clear lock buffer
`
`1\ 502
`
`' 504
`
`~,.
`
`Return
`
`Fig. 33
`
`NETAPP, INC. EXHIBIT 1015
`Page 43 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`"""' ~ c
`0 ......,
`N
`~
`~ ......
`'JJ. =-~
`
`N c c c
`"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`~
`I
`
`I
`
`y
`
`assert?
`
`Backoff signal )
`
`Fig. 34
`
`and state=M/MC
`
`Command=l/0 memory read
`
`invalidate and state=M/MC
`
`Command=cacheable read or read
`
`with state of cache line
`Assert snoop hit signal 1
`
`Command=l/0 memory read
`
`and state= E
`
`hit signal IJ
`I
`Activate H
`
`to 1/IWT
`
`Drive snoop
`16,
`
`backoff signal
`
`I
`
`. 508
`
`506
`
`address/command
`
`Decodes
`
`..L
`
`520
`
`memory write and state=M/MC
`
`Command=cacheable write or 1/0
`
`memorv write and state=E
`and state=E; Command=l/0
`Command=cacheable write
`
`invalidate and state=E
`
`Command=cacheable read or read
`
`invalidate and state=S/SWT
`
`Command=cacheable read or write
`
`CIU Interface Unit latches in
`
`from snoop path
`address/command
`
`521
`
`of cache line
`Change state
`
`1
`
`Return 11111
`
`NETAPP, INC. EXHIBIT 1015
`Page 44 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 43 of 160
`
`6,065,077
`
`Command=cacheable write and state=E;
`1 Command=l/0 memory write and state=E 1
`
`'522
`
`'s24
`
`Place device identifier on
`STARGET _ID signal
`
`Await deferred write reply
`from initiator interface device
`
`Respond to deferred write reply,
`obtain channel,
`write to external cache
`,,
`Return
`
`'528
`
`Fig. 35
`
`NETAPP, INC. EXHIBIT 1015
`Page 45 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 44 of 160
`
`6,065,077
`
`Command=cacheable write or 1/0 memory )
`1
`write and state=M/MC
`
`Place device identifier on
`
`STARGET_ID signal
`
`Await deferred write reply
`from initiator interface device
`
`'s3o
`
`'s32
`
`'534
`
`Response to deferred write reply,
`obtain channel, inquire internal cache
`and write to external cache
`
`' 536
`
`,,
`Return
`
`Fig. 36
`
`NETAPP, INC. EXHIBIT 1015
`Page 46 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 45 of 160
`
`6,065,077
`
`(
`
`Command=cacheable read
`or read invalidate and state=E
`,,.
`Transmit via channel to CIU
`read external cache command
`
`+
`
`Await deferred read reply
`from CIU
`
`,,
`
`Obtain communication path
`to initiator device
`,,
`
`Transmit deferred read to initiator
`device through communication path
`,,
`
`Change state of tag at target CIU
`
`,,
`
`Return
`
`Fig. 37
`
`538
`
`\
`
`540
`
`\ 542
`
`\ 544
`
`1\ 546
`
`\,
`548
`
`NETAPP, INC. EXHIBIT 1015
`Page 47 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 46 of 160
`
`6,065,077
`
`t
`
`Command=cacheable read
`or read invalidate and state=M/MC
`
`h
`550
`
`~,
`
`Transmit via channel to Cl U
`inquire internal cache,
`read external cache command
`
`+
`
`Await read data from Cl U
`
`~r
`
`Obtain communication path
`to initiator device
`
`~,
`
`Transmit deferred read to initiator
`device through communication path
`,
`
`Change state of tag at target CIU
`
`,,.
`Return
`
`Fig. 38
`
`1\
`552
`
`1\ 554
`
`1\
`556
`
`\
`
`558
`
`1\ 560
`
`NETAPP, INC. EXHIBIT 1015
`Page 48 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 47 of 160
`
`6,065,077
`
`h
`562
`
`' 564
`
`1\ 566
`
`[\
`568
`
`1\
`570
`
`~
`
`Command=l/0 memory read
`and state= E
`,,.
`
`Transmit via channel to CIU
`read external cache command
`
`,,.
`
`Await read data from CIU
`
`,,
`
`Obtain communication path
`to initiator device
`
`,,.
`Transmit deferred read to initiator
`device through
`communication path
`,,.
`Return
`
`Fig. 39
`
`NETAPP, INC. EXHIBIT 1015
`Page 49 of 191
`
`

`
`U.S. Patent
`
`May 16,2000
`
`Sheet 48 of 160
`
`6,065,077
`
`(
`
`Command=l/0 memory read
`and state=M/MC
`
`.,,.
`Transmit via channel to CIU
`inquire internal cache,
`read external cache command
`
`~,.
`
`Await read data from CIU
`
`,lr
`
`Obtain communication path
`to initiator device
`
`.,,.
`
`h
`572
`
`f\
`
`574
`
`1\
`576
`
`f\ 578
`
`Transmit deferred read to initiator
`device through
`communication path
`
`'\
`580
`
`~,.
`
`Return
`
`Fig. 40
`
`NETAPP, INC. EXHIBIT 1015
`Page 50 of 191
`
`

`
`.... =
`.... = 0\
`
`......::.
`......::.
`
`Ul
`
`0\
`
`'"""' ~ c
`0 ......,
`'0
`~
`~ .....
`'JJ. =(cid:173)~
`
`N c c c
`
`'"""' ~~
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`\Jl
`d •
`
`le
`
`Return
`
`596./
`
`Fig. 41
`
`path to MCU-IFU
`communication
`Obtain
`
`I 'lVI 1-\JCl\JI lvCU,
`
`+read
`
`110 bus
`BBU and place on
`Read data received at
`
`594./
`
`read data to BBU
`transmit though channel
`reply, obtain channel, and
`Respond to deferred read
`
`Await deferred read reply
`
`592./
`
`590./
`
`communication path
`and command on
`Transmit read address
`
`•
`•
`•
`•
`
`606'\ • •
`•
`+
`•
`1 BBU Interface unit receives transaction from channel I
`
`path to MCU-IFU 588.