FILE HISTORY
`US 6,633,945
`
`6,633,945
`PATENT:
`INVENTORS: Fu, Daniel
`Amdahl, Carlton T.
`Smith, III, Walstein Bennett
`
`TITLE:
`
`Fully connected cache coherent
`multiprocessing systems
`
`APPLICATION
`NO:
`FILED:
`ISSUED:
`
`US1999349641A
`
`08 JUL 1999
`14 OCT 2003
`
`COMPILED:
`
`25 AUG 2016
`
`NETAPP, INC. EXHIBIT 1012
`Page 1 of 198
`
`

`
`N .
`
`r-
`PATENT NUMBER
`
`8833946
`-illlM
`
`rr s-e
`
`a
`
`FILED WITH: E DISK (CRF) [: FICHE
`(Attached Ln pocket on right insideIap
`
`Certificate
`JAN 0 6 2004
`of Correction
`PREPARED AND APPROVED FOR ISSUE
`
`f i
`
`ORIGINAL
`
`ISSUING CLASSIFICATION
`CROSS REFERENCE(S)
`
`CLASS
`
`SUBCLASS
`
`CLASS
`
`SUBCLASS (ONE SUBCLASS PER BLOCK)
`
`INTERNATIONAL CLASSIFICATION
`
`-7
`
`,7' J
`
`)
`
`o +
`
`o
`
`sdo
`
`O Continued on Issue Slip Inside File Jacket
`
`TERMINAL
`DISCLAIMER
`
`Sheets Drwg.
`
`DRAWINGS
`Figs. Drwg.
`2
`
`Print Fig.
`
`CLAIMS ALLOWED
`Print Claim for O.G.
`Total Claims
`/0
`
`0 a) The term of this patentNOTICE
`(date)
`subsequent to
`has been disclaimed.
`
`;(
`
`Ag
`(Assistant Examiner)
`
`/f(0
`
`)
`
`NOTICE OF ALLOWANCE MAILED
`ALLOWANCE MAILED
`
`O b) The term of this patent shalt
`not extend beyond the expiration date
`of U.S Patent. No.
`
`'
`UMATI LEFKOWITZ
`PRIMARY EXAMINER " t~'
`
`F -/
`
`C)"g
`
`ISSUE FEE
`Date Paid
`Amount Due
`
`3
`
`_
`
`(Primary Examiner)
`
`(Date
`
`.//
`
`o C) The terminal ____months of
`this patent have been disclaimed.
`Illl1
`I
`
`(
`
`Examiner)
`I
`eg l InItruments Examiner)
`
`Dat
`
`ISSUE BATCH NUMBER
`
`__\
`
`.
`
`._
`
`WARNING:
`i
`..
`-
`_
`-.
`-
`The information disclosed herein may be restricted. Unauthized disclosure may be prohibited by the United states oae Title 35, Sections 12i, 1 8ano aneu.
`Possession outside the U.S. Patent & Trademark Office is restrilted to authorized employees and contractors only.
`Form PTO*436A
`(Rev. 6/986)
`
`-_"
`
`tJ
`
`wa a_
`
`I
`
`J Tal
`
`A
`
`L J~ w "A
`
`"'
`
`wJ QS
`
`ISc
`
`i- 'Grr ' FILE
`
`(LABEL AREA)
`
`-
`
`(FACE)
`
`'
`'
`- ~-"1"~11~~""~"~~~'~'~~"""'
`
`NETAPP, INC. EXHIBIT 1012
`Page 2 of 198
`
`

`
`6,633,945
`
`FULLY CONNECTED CACHE COHERENT MULTIPROCESSING
`SYSTEMS
`
`Transaction History
`
`
`Transaction Description
`Date
`Initial Exam Team nn
`07-15-1999
`IFW Scan & PACR Auto Security Review
`07-30-1999
`08-09-1999 Notice Mailed--Application Incomplete--Filing Date Assigned
`12-21-1999 Application Is Now Complete
`12-22-1999 Application Dispatched from OIPE
`02-14-2000 Case Docketed to Examiner in GAU
`10-10-2000 Case Docketed to Examiner in GAU
`05-25-2001 Correspondence Address Change
`05-25-2001 Change in Power of Attorney (May Include Associate POA)
`05-25-2001 Change in Power of Attorney (May Include Associate POA)
`07-17-2001 Change in Power of Attorney (May Include Associate POA)
`07-17-2001 Correspondence Address Change
`07-17-2001 Change in Power of Attorney (May Include Associate POA)
`12-10-2001 Case Docketed to Examiner in GAU
`12-17-2001 Quayle action
`12-18-2001 Mail Ex Parte Quayle Action (PTOL - 326)
`04-24-2002
`Information Disclosure Statement (IDS) Filed
`04-24-2002
`Information Disclosure Statement (IDS) Filed
`04-24-2002 Request for Continued Examination (RCE)
`04-24-2002 Request for Extension of Time - Granted
`04-24-2002 Workflow - Request for RCE - Begin
`04-30-2002 Date Forwarded to Examiner
`04-30-2002 Disposal for a RCE / CPA / R129
`07-15-2002 Mail Non-Final Rejection
`07-15-2002 Non-Final Rejection
`03-12-2003 Response after Non-Final Action
`03-17-2003 Date Forwarded to Examiner
`04-08-2003 Case Docketed to Examiner in GAU
`05-16-2003 Notice of Allowance Data Verification Completed
`05-16-2003 Case Docketed to Examiner in GAU
`05-19-2003 Mail Notice of Allowance
`05-21-2003 Dispatch to Publications
`05-23-2003 Receipt into Pubs
`05-28-2003 Workflow - File Sent to Contractor
`06-02-2003 Workflow - Drawings Finished
`06-02-2003 Workflow - Drawings Matched with File at Contractor
`06-02-2003 Workflow - Drawings Received at Contractor
`06-02-2003 Workflow - Drawings Sent to Contractor
`07-14-2003 Receipt into Pubs
`08-11-2003
`Issue Fee Payment Verified
`08-11-2003 ENTITY STATUS SET TO UNDISCOUNTED (INITIAL DEFAULT SETTING
`OR STATUS CHANGE)
`Issue Fee Payment Received
`08-11-2003
`08-27-2003 Application Is Considered Ready for Issue
`08-28-2003 Receipt into Pubs
`09-25-2003
`Issue Notification Mailed
`10-14-2003 Recordation of Patent Grant Mailed
`10-14-2003 Patent Issue Date Used in PTA Calculation
`12-09-2003 Post Issue Communication - Certificate of Correction
`10-28-2008 Change in Power of Attorney (May Include Associate POA)
`10-28-2008 Correspondence Address Change
`04-12-2016 File Marked Found
`
`NETAPP, INC. EXHIBIT 1012
`Page 3 of 198
`
`

`
`* 1
`
`093496i41
`
`1. Appicatio
`
`5.f~~
`
`aPers.
`
`wy/7!!ww
`
`CONTENTS
`Date received
`(Incl. C. of M.)
`or
`:.Date Mailed
`ii42.
`
`___________
`
`JUL ±W93~-
`
`Date received
`(Inca. C. of M.)
`or
`Date Mailed
`
`3.
`L ut
`
`1 44.
`
`45.
`
`.1.
`
`5
`
`.
`
`6.
`
`7.
`
`~zz.47. _____________
`
`____
`
`___
`
`____
`
`___
`
`48. _
`5%~C toI49.
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`10. S-abc
`
`rrt
`
`__
`
`±
`
`50.
`
`S r"51.
`
`---------
`
`-
`
`12.
`
`X13.
`
`4
`
`2
`
`53.
`
`_______
`
`_
`
`55. _
`56gA s.
`
`_
`
`_
`
`_
`
`_
`
`______
`
`-4
`
`____
`
`------
`
`231
`
`i.
`
`1
`
`.57.
`
`_ _ _ _ _ _ _
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`20.
`
`21..
`22.;:.
`
`241 V
`
`- -______,_
`
`t
`
`-7
`
`/60. _
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`61.
`
`62.
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`6_ 3.
`
`_
`
`_-
`
`_________'_
`
`64.
`65.
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`4
`
`_ _ _
`
`66. _
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`6..
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`25c
`
`26. _
`
`27. _
`
`28. _
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`68. _
`
`69. _
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`29. _
`
`30. __
`
`31. __
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_-
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`70.
`
`71.
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`32.
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`-
`
`_
`
`__
`
`_
`
`_
`
`72. _
`
`73. _
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`33.
`
`__
`
`_ __
`
`_ __
`
`_ __
`
`_ _74.
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`34. __ _ __
`
`_ __
`
`_ __
`
`_ _75.
`
`__
`
`_ _
`
`_ _ _
`
`_ _ _
`
`_ _
`
`_ _ _
`
`_ _ _
`
`_ _ 76.
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_ _ _ _
`
`_
`
`77. _
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`35.
`
`36. ____
`
`37. __
`
`38. _
`
`39.
`
`40. __
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`78.
`
`79.
`
`80. _
`
`81.
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`_
`
`NETAPP, INC. EXHIBIT 1012
`Page 4 of 198
`
`

`
`ISSUE SLIP STA.MLE AREA (for additiondi
`
`cross references)
`
`_
`
`POSITION
`
`--
`
`I
`
`I- -
`-INIAL
`
`,
`-
`
`- --
`
`"r---
`
`--
`
`.-
`
`-
`.. ,.
`
`ID NO.
`
`I
`
`-
`
`-
`
`DATE
`
`FEE DETE MINATION
`O.I.P.E. CLASSIFIER
`FORMALITY REVIEW
`
`--
`
`I
`
`~7
`
`1 I ,e Ic r D
`
`Ii
`
`I
`
`'
`INDEX OF CLAIMS
`Rejected
`.............................
`N ..............................
`Non-elected
`V
`Allowed
`I
`Interference
`.................................
`S ................................
`(Through numeral)... Canceled
`-
`A ..............................
`Appeal
`................................. Restricted
`0
`................................ Objected
`
`Date
`
`I
`Claim
`
`I
`
`Date
`
`I
`
`~
`
`.
`
`101
`
`1 1
`
`04
`105
`106
`107
`108
`09
`110 1U
`112
`
`11
`
`11
`115
`116
`11
`118
`
`121
`
`1 1
`
`2
`125
`
`127
`128
`
`31
`
`13
`
`13
`
`3 3
`
`8
`39
`140
`
`141
`14_
`14
`14
`45
`
`46
`147
`140
`14
`15--H...
`
`,
`
`59
`60
`61
`62
`63
`64
`65
`66
`67
`68
`69
`70
`71
`72
`73
`74
`75
`76
`
`77
`78
`79
`80
`81
`82
`83
`84
`85
`86
`
`87
`88
`89
`90
`91
`92
`93
`94
`
`96
`97
`
`98
`99
`
`Claim
`
`Date
`
`3 45 6 8
`
`Q' 9
`S10
`0 11
`
`y
`
`12
`13
`14
`15
`16
`17
`18
`19
`20
`211
`22_
`230
`24
`
`--
`
`l-
`
`:-
`
`I -
`
`-
`
`--.
`
`30 ..
`31 7
`32
`
`34
`35
`36
`37
`38
`39
`- ? -
`40
`41
`42
`43,
`44
`45
`46
`47
`48
`49
`50
`
`r r
`
`Claim
`
`I
`
`51
`52
`53..
`54
`55
`56
`57
`
`If more than 150 claims or 10 actions
`staple additional sheet here
`
`(LEFT INSIDE)
`
`I
`
`NETAPP, INC. EXHIBIT 1012
`Page 5 of 198
`
`

`
`SEARCH NOTES
`(INCLUDING SEARCH STRATEGY)
`
`i
`
`Exmr.
`
`Date
`
`w
`
`7
`
`INTERFERENCE SEARCHED
`Exmr.
`Date
`Sub.
`Class
`
`I
`
`'l
`
`i
`',o
`
`k r 3
`"
`,,
`
`.,
`
`,
`
`Su".
`
`uL Qc~i
`
`.
`
`--
`
`I
`
`(RIGHT OUTSIDE)
`
`i
`
`L--.___
`
`.
`
`.
`
`1
`
`NETAPP, INC. EXHIBIT 1012
`Page 6 of 198
`
`

`
`L Number
`1
`2
`3
`4
`5
`6
`7
`8
`9
`10
`11
`12
`13
`14
`15
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`24
`
`26
`
`25
`
`2
`
`1171
`
`119
`
`2
`
`119
`
`0
`
`19
`
`Hits Search Text
`1 6457087.pn.
`1 6466825.pn.
`1 6065077.pn.
`1 6292705.pn.
`1 6522701.pn.
`1 6516442.pn.
`171
`fcu
`1665 smp
`5
`fcu and smp
`801546 switch matrix or switch fabric
`smp and (switch matrix or switch fabric)
`857
`fcu and (switch matrix or switch fabric)
`101
`(mutiple or plurality) with fcu
`8
`(mutiple or plurality or multi) adj processor$1
`8016
`433272 shared memory
`((mutiple or plurality or multi) adj processor$1) with (shared
`2663
`memory)
`(fcu and (switch matrix or switch fabric)) and (((mutiple or
`plurality or multi) adj processor$1) with (shared memory))
`(switch matrix or switch fabric) and (((mutiple or plurality or
`multi) adj processor$1) with (shared memory))
`2867 7and (((mutiple or plurality or multi) adj processor$1) with
`(shared memory))
`(smp and (switch matrix or switch fabric)) and (7and (((mutiple USPAT
`or plurality or multi) adj processor$1) with (shared memory)))
`fcu and (((mutiple or plurality or multi) adj processor$1) with
`(shared memory))
`(smp and (switch matrix or switch fabric)) and ((switch matrix
`or switch fabric) and (((mutiple or plurality or multi) adj
`processor$1) with (shared memory)))
`1691271 pp or point to point
`((smp and (switch matrix or switch fabric)) and ((switch matrix
`106
`or switch fabric) and (((mutiple or plurality or multi) adj
`processor$1) with (shared memory)))) and (pp or point to
`point)
`(((smp and (switch matrix or switch fabric)) and ((switch matrix USPAT
`or switch fabric) and (((mutiple or plurality or multi) adj
`processor$1) with (shared memory)))) and (pp or point to
`point)) and nglo
`ngio
`
`DB
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`USPAT
`
`USPAT
`
`USPAT
`
`USPAT
`
`USPAT
`
`USPAT
`
`USPAT
`USPAT
`
`USPAT
`
`Time stamp
`2003/05/16 08:24
`2003/05/16 08:26
`2003/05/16 08:27
`2003/05/16 08:27
`2003/05/16 08:28
`2003/05/16 08:31
`2003/05/16 08:34
`2003/05/16 08:34
`2003/05/16 08:35
`2003/05/16 08:36
`2003/05/16 08:36
`2003/05/16 08:37
`2003/05/16 08:43
`2003/05/16 08:44
`2003/05/16 08:45
`2003/05/16 08:45
`
`2003/05/16 08:46
`
`2003/05/16 08:46
`
`2003/05/16 08:48
`
`2003/05/16 08:49
`
`2003/05/16 08:48
`
`2003/05/16 08:49
`
`2003/05/16 08:50
`2003/05/16 08:50
`
`2003/05/16 08:50
`
`2003/05/16 08:50
`
`Search History 5/16/03 8:58:19 AM Page 1
`C:\APPS\EAST1Workspaces\swfabricproc.sharedmemory.wsp
`
`~"II~~'~~"~'~"~"""'~"~~"'~~
`
`NETAPP, INC. EXHIBIT 1012
`Page 7 of 198
`
`

`
`RAM Fee History
`
`http://ram.uspto.gov:8888/cgi-bin/fee_history.cgi?name number09349641
`
`YYYYIY- -
`
`III"~--~"YYYI""""-
`
`Revenue Accounting and Management
`
`Name/Number: 09349641
`Start Date: Any Date
`
`Total Records Found: 10
`End Date: Any Date
`
`Fee
`Code
`201
`205
`216
`581
`581
`116
`126
`179
`1501
`8001
`
`Fee Amount Mailroom Date
`
`Payment Method
`
`$380.00
`$65.00
`$190.00
`$40.00
`$40.00
`$400.00
`$180.00
`$740.00
`$1,300.00
`$30.00
`
`12/13/1999
`12/13/1999
`12/13/1999
`06/27/2000
`04/09/2001
`04/24/2002
`04/24/2002
`04/24/2002
`08/11/2003
`08/11/2003
`
`DA 500689
`DA 500689
`DA 500689
`DA 500689
`CK
`CC
`CC
`CC
`CC
`CC
`
`Tran
`Type
`
`I1 1 1 1 1 1 1
`
`1 I
`
`1
`
`Accounting
`Date
`12/15/1999
`12/15/1999
`12/15/1999
`07/19/2000
`04/20/2001
`04/25/2002
`04/25/2002
`04/25/2002
`08/12/2003
`08/12/2003
`
`Sequence
`Num.
`00000273
`00000274
`00000275
`00000029
`00000250
`00000002
`00000003
`00000001
`00000375
`00000376
`
`8/25/03 1:57 PM
`
`i
`
`-----
`
`---LI-l--- i_-l__
`
`-- --l *-L _
`
`
`
`--~ ~YY---Y I- Y--
`
`L I- YI Y
`
`
`
`ILL~YYLlrYI--- -- --
`
`Fe e History
`Query
`
`NETAPP, INC. EXHIBIT 1012
`Page 8 of 198
`
`

`
`(12) United States Patent
`Fu et al.
`
`(54) FULLY CONNECTED CACHE COHERENT
`MULTIPROCESSING SYSTEMS
`
`(75)
`
`Inventors: Daniel Fu, Sunnyvale, CA (US);
`Carlton T. Amdahl, Alameda County,
`CA (US); Walstein Bennett Smith, III,
`Palo Alto, CA (US)
`
`(73) Assignee: Conexant Systems, Inc., Newport
`Beach, CA (US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`Appl. No.: 09/349,641
`
`Filed:
`
`Jul. 8, 1999
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 09/281,749, filed on
`Mar. 30, 1999, now Pat. No. 6,516,442, which is a continu-
`ation-in-part of application No. 09/163,294, filed on Sep. 29,
`1998, now Pat. No. 6,292,705, which is a continuation-in-
`part of application No. 08/986,430, filed on Dec. 7, 1997,
`now Pat. No. 6,065,077.
`
`. .. .. .
`
`... .. ... .
`
`Int. Cl. 7 .. .. .....................
`G06F 13/00
`(51)
`(52) U.S. Cl ...................... 710/316; 710/317; 710/29;
`709/213; 711/130
`(58) Field of Search .............................. 710/100, 305,
`710/313, 315, 316, 317, 29; 711/144, 143,
`130, 147; 709/213
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,315,308 A
`4,438,494 A
`4,480,307 A
`5,161,156 A
`5,271,000 A
`5,313,609 A
`5,335,335 A
`5,440,698 A
`5,505,686 A
`
`2/1982
`3/1984
`10/1984
`11/1992
`12/1993
`5/1994
`8/1994
`8/1995
`4/1996
`
`Jackson
`Budde et al.
`Budde et al.
`Baum et al.
`Engbersen et al.
`Baylor et al.
`Jackson et al.
`Sindhu et al.
`Willis et al.
`
`11111111|1111 1|111
`||U S00 33 45B111
`|u111 11111
`US 6,633,945 B1
`Oct. 14, 2003
`
`(10) Patent No.:
`(45) Date of Patent:
`
`5,511,226
`5,513,335
`5,524,234
`5,526,380
`5,535,363
`5,537,569
`5,537,575
`5,553,310
`5,561,779
`5,568,620
`5,574,868
`5,577,204
`5,581,729
`5,588,131
`5,594,886
`5,602,814
`5,606,686
`5,634,043
`5,634,068
`5,644,754
`5,655,100
`5,657,472
`
`4/1996
`4/1996
`6/1996
`6/1996
`7/1996
`7/1996
`7/1996
`9/1996
`10/1996
`10/1996
`11/1996
`11/1996
`12/1996
`12/1996
`1/1997
`2/1997
`2/1997
`5/1997
`5/1997
`7/1997
`8/1997
`8/1997
`
`Zilka
`McClure
`Martinez, Jr. et al.
`Izzard
`Prince
`Masubuchi
`Foley
`Taylor et al.
`Jackson
`Sarangdhar et al.
`Marisetty
`Brewer et al.
`Nishtala et al.
`Borrill
`Smith et al.
`Jaquette et al.
`Tarui et al.
`Self et al.
`Nishtala et al.
`Weber
`Ebrahim et al.
`Van Loo et al.
`
`(List continued on next page.)
`
`OTHER PUBLICATIONS
`
`Technical White Paper, Sun TM Enterprise TM 10000
`Server, Sun Microsystems, Sep. 1998.
`Alan Charlesworth, Starfire: Extending the SMP Envelope,
`IEEE Micro, Jan./Feb. 1998, pp. 39-49.
`
`(List continued on next page.)
`Primary Examiner-Sumati Lefkowitz
`Assistant Examiner-X. Chung-Trans
`(74) Attorney, Agent, or Firm-Keith Kind; Kelly H. Hale
`(57)
`ABSTRACT
`
`Fully connected multiple FCU-based architectures reduce
`requirements for Tag SRAM size and memory read laten-
`cies. A preferred embodiment of a symmetric multiprocessor
`system includes a switched fabric (switch matrix) for data
`transfers that provides multiple concurrent buses that enable
`greatly increased bandwidth between processors and shared
`memory. A high-speed point-to-point Channel couples com-
`mand initiators and memory with the switch matrix and with
`I/O subsystems.
`
`10 Claims, 15 Drawing Sheets
`
`MPC
`
`PU BUS
`
`NETAPP, INC. EXHIBIT 1012
`Page 9 of 198
`
`

`
`US 6,633,945 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,682,516 A
`5,684,977 A
`5,696,910 A
`5,796,605 A
`5,829,034 A
`5,895,495 A
`5,897,656 A
`5,940,856 A
`5,946,709 A
`5,978,411 A
`6,044,122 A
`6,065,077 A *
`6,125,429 A *
`6,145,007 A
`6,279,084 B1
`6,289,420 B1 *
`6,292,705 B1
`6.295.581 B1 *
`
`10/1997
`11/1997
`12/1997
`8/1998
`10/1998
`4/1999
`4/1999
`8/1999
`8/1999
`11/1999
`3/2000
`5/2000
`9/2000
`11/2000
`8/2001
`9/2001
`9/2001
`9/2001
`
`Sarangdhar et al.
`Van Loo et al.
`Pawlowski
`Hagersten
`Hagersten et al.
`Arimilli et al.
`Vogt et al.
`Arimilli et al.
`Arimilli et al.
`Kitade et al.
`Ellersick et al.
`Fu .......................... 710/100
`Goodwin et al. ........ 711/143
`Dokic et al.
`VanDoren et al.
`Cypher ....................... 711/144
`Wang et al.
`DeRoo ....................... 711/135
`
`OTHER PUBLICATIONS
`
`Joseph Heinrich, Origin TM and Onyz2 TM Theory of
`Operations Manual, Document No. 007-3439-002, Silicon
`Graphics, Inc., 1997.
`White Paper, Sequent's NUMA-Q SMP Architecture,
`Sequent, 1997.
`White Paper, Eight-way Multiprocessing, Hewlett-Packard,
`Nov. 1997.
`George White & Pete Vogt, Profusion, a Buffered, Cache-
`Coherent Crossbar Switch, presented at Hot Interconnects
`Symposium V, Aug. 1997.
`Alan Charlesworth, et al., Gigaplane-XP: Extending the
`Ultra Enterprise Family, presented at Hot Interconnects
`Symposium V, Aug. 1997.
`
`James Loudon & Daniel Lenoski, The SGI Origin: A
`ccNUMA Highly Scalable Server, Silicon Graphics, Inc.,
`presented at the Proc. Of the 2 4 h Int'l Symp. Computer
`Architecture, Jun. 1997.
`Mike Galles, Spider: A High-Speed Network Interconnect,
`IEEE Micro, Jan./Feb. 1997, pp. 34-39.
`T.D. Lovett, R. M. Clapp and R. J. Safranek, Numa-Q: an
`SCI-based Enterprise Server, Sequent, 1996.
`Daniel E. Lenoski & Wolf-Dietrich Weber, Scalable Shared-
`Memory Multiprocessing, Morgan Kaufmann Publishers,
`1995, pp. 143-159.
`David B. Gustavson, The Scalable coherent Interface and
`Related Standards Projects, (as reprinted in Advanced Mul-
`timicroprocessor Bus Architectures, Janusz Zalewski, IEEE
`computer Society Press, 1995, pp. 195-207.).
`Kevin Normoyle, et al., UltraSPARC TM Port Architecture,
`Sun Microsystems, Inc., presented at Hot Interconnects III,
`Aug. 1995.
`Kevin Normoyle, et al., UltraSPARC TM Port Architecture,
`Sun Microsystems, Inc., presented at Hot Interconnects III,
`Aug. 1995, UltraSparc Interfaces.
`Kai Hwang, Advanced Computer Architecture: Parallelism,
`Scalability, Programmability, McGraw-Hill, 1993, pp.
`355-357.
`Jim Handy, The Cache Memory Book, Academic Press,
`1993, pp. 161-169.
`
`Angel L. Decegama, Parallel Processing Architectures and
`VLSI Hardware, vol. 1, Prentice-Hall, 1989, pp. 341-344.
`
`* cited by examiner
`
`NETAPP, INC. EXHIBIT 1012
`Page 10 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 1 of 15
`
`US 6,633,945 B1
`
`A
`
`/t=
`
`On
`
`n
`
`A
`
`w
`
`o,
`
`Sm
`06
`
`V
`
`4-
`
`4-
`
`-o
`uJ
`
`t-m
`
`V
`
`OT-
`
`-
`
`CIL
`U#
`
`LU
`
`0-
`
`0C-
`
`NETAPP, INC. EXHIBIT 1012
`Page 11 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 2 of 15
`
`US 6,633,945 B1
`
`o
`
`W
`
`00
`rr €)
`Co
`r
`w
`2
`
`NETAPP, INC. EXHIBIT 1012
`Page 12 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 3 of 15
`
`US 6,633,945 B1
`
`- o
`
`INTERFACE
`SICONTROL
`
`INTERFACE
`CONTROL
`
`0LC)
`
`CIB
`
`CIB
`
`4
`
`!
`
`---
`
`-CIB-I-B--------
`N
`CIB
`,
`
`---
`
`CIB
`
`,
`
`CIB
`
`CIB
`
`0
`
`I
`
`0 ,-<%--r l -II
`
`--
`m
`
`I
`
`I 1Ui
`
`,LL
`
`',,
`I
`
`-- I
`( I
`
`-
`
`I a-
`
`0 -II- a
`
`-
`
`-
`
`eo
`
`e
`
`-
`
`a
`
`,
`I
`
`.
`
`e
`
`0,,
`
`r->
`-- -r--L--
`
`a
`
`0
`
`I
`
`I
`
`II
`
`.
`
`- -
`
`- -
`
`a-----------
`SI
`
`'-
`
`I
`
`---- -F-0
`
`I W
`
`_-
`0
`
`I-,
`I -
`I
`I, I
`
`'u
`
`N
`
`I
`,
`
`C7
`L
`
`CONTROLLER
`
`I
`I
`
`I
`
`m
`Ico -- z
`I-------
`
`1
`,
`
`o
`
`I---
`
`Q
`
`I
`
`i - ..... . . . , --- - - - - - - - - - - -- - - - - - .
`
`.
`
`.
`
`.
`
`IC
`
`.
`
`I ' L------------
`
`co
`
`~Z
`
`NETAPP, INC. EXHIBIT 1012
`Page 13 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 4 of 15
`
`US 6,633,945 B1
`
`m
`
`m
`
`m
`
`m
`
`m
`
`NETAPP, INC. EXHIBIT 1012
`Page 14 of 198
`
`

`
`U. S. Patent
`
`Oct. 14, 2003
`
`Sheet 5 of 15
`
`US 6,633,945 Bi
`
`in
`
`Co
`Cf)
`
`C
`
`)
`
`U-
`uCD)o
`
`'I
`
`I-
`
`UJ
`0W
`
`WJU}
`
`U
`Q'
`a.
`
`of
`
`z
`
`~
`W
`
`m
`
`00
`
`Q n
`C/jC
`
`p
`w ¢
`
`LL
`
`a_
`
`U
`
`M W
`coo
`
`CML
`
`M N
`
`m' co
`
`-
`
`It)
`
`'-C"
`
`0)O
`
`Fl
`
`161
`
`1 im
`m
`
`EMT
`--m
`t
`
`lii
`
`liii
`
`lIi
`
`II
`
`lii
`
`II
`
`1
`
`1
`
`H
`
`t
`
`U)
`
`CO
`
`} O
`
`0cA
`
`Co,
`U
`
`I-a-
`
`asa-
`
`U i
`
`p Id:
`_ c
`
`NETAPP, INC. EXHIBIT 1012
`Page 15 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 6 of 15
`
`US 6,633,945 B1
`
`.o
`
`0l
`fo
`(0.
`
`a(
`
`NETAPP, INC. EXHIBIT 1012
`Page 16 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 7 of 15
`
`US 6,633,945 B1
`
`m
`
`m
`
`mmaC-)
`
`mm
`
`m
`
`a1n&n1in1i
`
`00101D III I
`
`UJUJ ILUI
`O 0 10 0
`
`F1F0
`hlcJlO
`
`U
`
`U
`
`~
`
`U
`
`~i
`
`II
`
`L
`
`NETAPP, INC. EXHIBIT 1012
`Page 17 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 8 of 15
`
`US 6,633,945 B1
`
`----------- i
`
`ii i
`
`i ii
`
`_____ - - - - - - - - - - -
`
`- - - - -
`
`-------
`
`J
`
`4-
`
`NETAPP, INC. EXHIBIT 1012
`Page 18 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 9 of 15
`
`US 6,633,945 B1
`
`m
`o
`
`3~a::
`0
`
`>m-
`0I
`
`m
`0
`
`f III
`n n
`I II
`
`-CD
`Co
`
`5a
`
`CO
`co
`
`a
`
`vC
`coI
`
`u
`
`mm
`
`H
`
`)
`LL
`
`Co
`
`1a
`
`0"
`
`.
`
`0 0
`
`Co
`Co
`
`a-
`
`-I
`
`mm
`
`H
`
`Qo
`Co
`Co
`
`Ua
`
`NETAPP, INC. EXHIBIT 1012
`Page 19 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 10 of 15
`
`US 6,633,945 B1
`
`NETAPP, INC. EXHIBIT 1012
`Page 20 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 11 of 15
`
`US 6,633,945 B1
`
`d
`
`rz
`
`NETAPP, INC. EXHIBIT 1012
`Page 21 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 12 of 15
`
`US 6,633,945 B1
`
`Nj
`
`LL
`
`Cn)
`
`Co
`
`U
`
`a.
`
`Co
`
`m
`
`Co
`
`a
`
`Co
`
`Co
`
`a
`
`Co
`
`a
`
`ak-
`Co
`
`Co
`5
`
`C)
`
`m aO
`
`aw a
`~O Q
`a U
`
`NETAPP, INC. EXHIBIT 1012
`Page 22 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 13 of 15
`
`US 6,633,945 B1
`
`CoU
`
`mC
`
`CD
`
`Ua U
`
`)
`
`m U
`
`m
`
`SCn
`cn
`U)
`
`mUa
`
`U da
`
`nn
`
`a
`
`cn
`Co
`U
`
`a
`
`om aMa
`
`-
`
`NETAPP, INC. EXHIBIT 1012
`Page 23 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 14 of 15
`
`US 6,633,945 B1
`
`m
`C
`
`ad
`NC
`
`N,
`
`N C
`
`-
`
`C 4
`
`L Uf
`
`c
`0
`
`- It
`
`I NC
`
`YUO m
`
`o
`
`NETAPP, INC. EXHIBIT 1012
`Page 24 of 198
`
`

`
`U.S. Patent
`
`Oct. 14, 2003
`
`Sheet 15 of 15
`
`US 6,633,945 B1
`
`oi
`
`*----a,
`
`I I II I I
`
`I--
`
`I--
`I
`
`I--
`3
`
`I
`
`--
`
`-I
`
`--
`
`I III I I I I II II
`
`I
`I
`
`II
`
`II I I
`
`I
`
`I
`
`I
`
`*----4,
`
`1~
`
`*----4,
`
`*----I,
`
`*----I
`
`C-)
`
`-'-,
`
`1---
`1
`
`1U
`
`I
`
`1--
`1I
`I U
`1
`
`1,,,
`
`11 1 1111 1 111 1
`
`w
`
`111
`
`I1 1
`
`111 1
`
` L
`
`W
`
`U
`
`1 1
`
`1
`
`'
`
`111 11
`
`1 1111r 11111 1 1 111
`
`---
`
`I1I1 I I1I i
`
`11
`
`11 1 11
`
`IIII I1 I I I I I 1
`
`NETAPP, INC. EXHIBIT 1012
`Page 25 of 198
`
`

`
`US 6,633,945 B1
`
`cies. A preferred embodiment of a symmetric multiprocessor
`system includes a switched fabric (switch matrix) for data
`transfers that provides multiple concurrent buses that enable
`greatly increased bandwidth between processors and shared
`memory. A high-speed point-to-point Channel couples com-
`mand initiators and memory with the switch matrix and with
`I/O subsystems.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`FULLY CONNECTED CACHE COHERENT
`MULTIPROCESSING SYSTEMS
`
`CROSS-REFERENCE TO RELATED APPLICATIONS
`This patent application is a continuation-in-part of the 5
`following commonly-owned, U.S. patent application Ser.
`Nos.: U.S. application Ser. No. 08/986,430 now U.S. Pat.
`No. 6,065,077, AN APPARATUS AND METHOD FOR A
`CACHE COHERENT SHARED MEMORY MULTIPRO-
`CESSING SYSTEM filed Dec. 7, 1997; U.S. application 10
`Ser. No. 09/163,294 now U.S. Pat. No. 6,292,705,
`METHOD AND APPARATUS FOR ADDRESS
`TRANSFERS, SYSTEM SERIALIZATION, AND CEN-
`TRALIZED CACHE AND TRANSACTION CONTROL,
`IN A SYMMETRIC MULTIPROCESSOR SYSTEM, filed
`Sep. 29, 1998; and U.S. application Ser. No. 09/281,749 now
`U.S. Pat. No 6,516,442, CACHE INTERFACE AND PRO-
`TOCOLS FOR CACHE COHERENCY IN A SCALABLE
`SYMMETRIC MULTIPROCESSOR SYSTEM, filed Mar.
`30, 1999; all of which are incorporated by reference herein. 20
`
`BACKGROUND
`
`FIG. 1 is a drawing of a prior-art generic symmetric
`shared-memory multiprocessor system using a shared-bus.
`FIG. 2 is a drawing of a preferred embodiment symmetric
`shared-memory multiprocessor system using a switched
`fabric data path architecture centered on a Flow-Control
`s15
`Unit (FCU).
`FIG. 3 is a drawing of the switched fabric data path
`architecture of FIG. 2, further showing internal detail of an
`FCU having a Transaction Controller (TC), Transaction Bus
`(TB), and Transaction Status Bus (TSB) according to the
`present invention.
`FIG. 4 is a drawing of a variation the embodiment of FIG.
`2, it which each CPU has its own CCU, and in which the
`channel interface and control is abstractly represented as
`being composed of a physical (PHY) link layer and a
`transport layer.
`FIG. 5 is a timing diagram comparing the memory trans-
`action performance of a system based on a flow control unit
`according to the present invention and a prior art shared-bus
`system.
`FIG. 6 is another view of the embodiment of FIG. 4.
`FIG. 7 is a drawing of a number of system embodiments
`according to the present invention. FIG. 7a illustrates a
`minimal configuration, 7b illustrates a 4-way configuration,
`7c illustrates an 8-way high-performance configuration, and
`7d illustrates a configuration for I/O intensive applications.
`FIG. 8 is a drawing of a CPU having an integral CCU.
`FIG. 9 illustrates a variation of the embodiment of FIG. 6
`using the integrated CPU/CCU of FIG. 8.
`FIGS. 10a-d illustrates variations of the embodiments of
`FIG. 7 using the integrated CPU/CCU of FIG. 8.
`FIG. 11 is a drawing of an 4-way embodiment of the
`present invention that includes coupling to an industry
`standard switching fabric for coupling CPU/Memory com-
`plexes with I/O devices.
`FIG. 12 is a drawing of an FCU-based architecture
`according to a first embodiment.
`FIG. 13 is a drawing of an FCU-based architecture
`according to a second embodiment.
`FIG. 14 defines the cache line characteristics of the
`systems of FIGS. 12 and 13.
`FIG. 15 defines the cache line definition.
`
`25
`
`30
`
`40
`
`FIGS. 2-11 show point to point cache coherent switch
`solution for multiprocessor systems that are the subject of
`copending and coassigned applications.
`Depending on the implementation specifics, these designs
`may be problematic in two respects:
`1. Tag SRAM size is expensive
`2. Latency is greater than desired
`First, SRAM Size Issue:
`To support L2 size=4 MB, total 64 GB memory and 64
`byte line size
`the TAG array entry will be 4 MB/64 Byte=64K entries
`the TAG size will be 14 bits
`The total TAG array size=14 bits *64K=917,504 bit/per
`CPU
`To support 8-way system, a duplicated TAG array size
`will be 8*14 bits *64K-about 8M bit SRAM.
`8 Mbit SRAM is too large for single silicon integrait even
`with 0.25 micron CMOS process.
`Second, Latency Issue:
`Although the switch fabric solutions of FIGS. 2-11 pro-
`vide scalability in memory throughput, maximum transac- 45
`tion parallelism, and easy PCB broad routing, the latency for
`memory read transactions is greater than desired.
`Example for Memory Read Transactions:
`CPU read transaction will first latched by CCU, CCU
`format transaction into channel command, CCU will send 5so
`the transaction through channel, FCU's IIF unit will
`de-serialize the channel command or data and perform cache
`coherency operation, then FCU will send the memory read
`transaction to MCU. MCU will de-serialize the channel
`command, send the read command to DRAM address bus, 5ss
`MCU read from DRAM data bus, send the data to FCU via
`channel, FCU will send data to CCU via channel. Finally the
`data is presented at CPU bus. A transaction for read crosses
`the channel four times. Each crossing introduces additional
`latency. What is needed is an SMP architecture with the 60
`benefits of the present FCU architecture, but with reduced
`Tag SRAM size requirements per chip and with reduced
`latencies.
`
`DETAILED DESCRIPTION
`
`System Overview
`
`FIG. 2 is a drawing of a preferred embodiment symmetric
`shared-memory multiprocessor system using a switched
`fabric data path architecture centered on a Flow-Control
`Unit (FCU) 220. In the illustrated embodiment, eight pro-
`cessors 120 are used and the configuration is referred herein
`as an "8P" system.
`The FCU (Flow Control Unit) 220 chip is the central core
`of the 8P system. The FCU internally
`implements a
`switched-fabric data path architecture. Point-to-Point (PP)
`
`SUMMARY
`
`65
`
`Fully connected multiple FCU-based architectures reduce
`requirements for Tag SRAM size and memory read laten-
`
`NETAPP, INC. EXHIBIT 1012
`Page 26 of 198
`
`

`
`US 6,633,945 B1
`
`maps from Channel Protocol addresses to targets. The
`mapping generally does not change or permute addresses.
`
`Summary of Key Components
`
`interconnect 112, 113, and 114 and an associated protocol
`define dedicated communication channels for all FCU I/O.
`The terms Channels and PP-Channel are references to the
`FCU's PP I/O. The FCU provides Point-to-Point Channel
`interfaces to up to ten Bus Bridge Units (BBUs) 240 and/or
`CPU Channel Units (CCUs, also known as Chanel Interface
`Units or CIUs) and one to four Memory Control Units
`(MCUs) 230. Two of the ten Channels are fixed to connect
`to BBUs. The other eight Channels can connect to either
`BBUs or CCUs. In an illustrative embodiment the number of
`CCUs is eight. In one embodiment the CCUs are packaged
`as a pair referred herein as a Dual CPU Interface Unit
`(DCIU) 210. In the 8P system shown, the Dual CPU
`Interface Unit (DCIU) 210 interfaces two CPUs with the
`FCU. Throughout this description, a reference to a "CCU"
`is understood to describe the logical operation of each half
`of a DCIU 210 and a references to "CCUs" is understood to
`apply to equally to an implementation that uses either single
`CCUs or DCIUs 210. CCUs act as a protocol converter
`between the CPU bus protocol and the PP-Channel protocol.
`The FCU 210 provides a high-bandwidth and low-latency
`connection among these components via a Data Switch, also
`referred herein as a Simultaneous Switched Matrix (SSM),
`or switched fabric data path. In addition to connecting all of
`these components, the FCU provides the cache coherency
`support for the connected BBUs and CCUs via a Transaction
`Controller and a set of cache-tags duplicating those of the
`attached CPUs' L2 caches. FIG. 5 is a timing diagram
`comparing the memory transaction performance of a system
`based on a flow control unit according
`to the present
`invention and a prior art shared-bus system.
`In a preferred embodiment, the FCU provides support two
`dedicated BBU channels, four dedicated MCU channels, up
`to eight additional CCU or BBU channels, and PCI peer-
`to-peer bridging. The FCU contains a Transaction Controller
`(TC) with reflected L2, states. The TC supports up to 200M
`cache-coherent transactions/second, MOSEI and MESI
`protocols, and up to 39-bit addressing. The FCU contains the
`Simultaneous Switch Matrix (SSM) Dataflow Switch, which
`supports non-blocking data transfers.
`In a preferred embodiment, the MCU supports flexible
`memory configurations, including one or two channels per
`MCU, up to 4 Gbytes per MCU (maximum of 16 Gbytes per
`system), with one or two memory banks per MCU, with one
`to four DIMMS per bank, of SDRAM, DDR-SDRAM, or
`RDRAM, and with non-interleaved or interleaved operation.
`In a preferred embodiment, the BBU supports both 32 and
`64 bit PCd bus configurations, including 32 bit/33 MHz, 32
`bit/66 MHz, and 64 bit/66 MHz. The BBU is also 5V
`tolerant and supports AGP.
`All connections between components occur as a series of
`"transactions." A transaction is a Channel Protocol request
`command and a corresponding Channel Protocol reply. For
`example, a processor, via a CCU, can perform a Read
`request that will be forwarded, via the FCU, to the MCU; the
`MCU will return a Read reply, via the FCU, back to the same
`processor. A Transaction Protocol Table (TPT) defines the
`system-wide behavior of every type of transaction and a
`Point-to-Point Channel Protocol defines the command for-
`mat for transactions.
`The FCU assumes that initiators have converted addresses
`from other formats to conform with the PP-Channel defini-
`tions. The FCU does do target detection. Specifically, the
`FCU determines the correspondence between addresses and
`specific targets via address mapping tables. Note that this
`mapping hardware (contained in the CFGIF and the TC)
`
`5
`
`25
`
`Transaction Controller (TC) 400. The most critical coher-
`ency principle obeyed by the FCU is the concept of a single,
`system-serialization point. The system-serialization point is
`the "funnel" through which all transactions must pass. By
`guaranteeing that all transactions pass through the system-
`10 serialization point, a precise order of transactions can be
`defined. (And this in turn implies a precise order of tag state
`changes.) In the FCU, the system-serialization point is the
`Transaction Controller (TC). Coherency state is maintained
`by the duplicate set of processor L2 cache-tags stored in the
`15 TC.
`The Transaction Controller (TC) acts as central system-
`serialization and cache coherence point, ensuring that all
`transactions in the system happen in a defined order, obeying
`defined rules. All requests, cacheable or not, pass through
`20 the Transaction Controller. The TC handles the cache coher-
`ency protocol using a duplicate set of L2 cache-tags for each
`CPU. It also controls address mapping inside the FCU,
`dispatching each transaction request to the appropriate target
`interface.
`Transaction Bus (TB) 3104 and Transaction Status Bus
`(TSB) 3106. All request commands flow through the Trans-
`action Bus. The Transaction Bus is designed to provide fair
`arbitration between all transaction sources (initiators) and
`the TC; it provides an inbound path to the TC, and distrib-