InfiniBandInfiniBandTMTMTMTM Architecture Architecture
`
`InfiniBandInfiniBand
`
` Architecture Architecture
`
`Specification Volume 1Specification Volume 1
`
`Specification Volume 1Specification Volume 1
`
`
` Release 1.0.a Release 1.0.a
`
` Release 1.0.a Release 1.0.a
`
`June 19, 2001
`Final
`
`Copyright © 1999, 2001, by InfiniBandSM Trade Association.
`All rights reserved.
`
`1
`
`APPLE 1029
`Apple et al. v. Memory Integrity
`IPR2015-00163
`
`

`
`1234567891
`
`0
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`21
`22
`23
`24
`25
`26
`27
`28
`29
`30
`31
`32
`33
`34
`35
`36
`37
`38
`39
`40
`41
`42
`
`InfiniBandTM Architecture Release 1.0.a
`VOLUME 1 - GENERAL SPECIFICATIONS
`
`June 19, 2001
`FINAL
`
`Table 1 Revision History
`
`Revision
`
`Release Date
`
`1.0
`
`1.0.a
`
`9/26/2000
`
`Release 1.0
`
`6/19/2001
`
`Release 1.0 augmented with errata material. Updates only
`correct errors - no additional features have been added.
`
`LEGAL DISCLAIMER
`
`"This version of a proposed IBTA specification is pro-
`vided "AS IS" and without any warranty of any kind,
`including, without limitation, any express or implied
`warranty of non-infringement, merchantability or fit-
`ness for a particular purpose.
`
`In no event shall IBTA or any member of IBTA be liable
`for any direct, indirect, special, exemplary, punitive,
`or consequential damages, including, without limita-
`tion, lost profits, even if advised of the possibility of
`such damages."
`
`InfiniBandSM Trade Association
`
` Page 2
`
`2
`
`

`
`1234567891
`
`0
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`21
`22
`23
`24
`25
`26
`27
`28
`29
`30
`31
`32
`33
`34
`35
`36
`37
`38
`39
`40
`41
`42
`
`InfiniBandTM Architecture Release 1.0.a
`VOLUME 1 - GENERAL SPECIFICATIONS
`
`TABLE OF CONTENTS
`
`June 19, 2001
`FINAL
`
`1.3
`1.4
`
`...................................................................................... 38
`Introduction
`Chapter 1:
`1.1
`Acknowledgments......................................................................................................... 38
`1.2
`InfiniBand Conceptual Overview .................................................................................. 40
`1.2.1
`The Problem .............................................................................................................. 40
`1.2.2
`Features .................................................................................................................... 41
`1.2.3
`Benefits...................................................................................................................... 41
`Scope ........................................................................................................................... 42
`Document Organization................................................................................................ 43
`1.4.1
`Series of Volumes...................................................................................................... 43
`1.4.2
`Volume 1 Organization .............................................................................................. 44
`Document Conventions ................................................................................................ 44
`1.5.1
`Byte Ordering ............................................................................................................ 44
`1.5.2
`Numeric Values.......................................................................................................... 45
`1.6
`Disclaimer..................................................................................................................... 46
`Chapter 2:
`Glossary
`............................................................................................ 47
`
`1.5
`
`3.2
`
`3.3
`3.4
`
`Architectural Overview .................................................................... 61
`Chapter 3:
`3.1
`Architecture Scope ....................................................................................................... 62
`3.1.1
`Topologies & Components......................................................................................... 63
`Communication............................................................................................................. 65
`3.2.1
`Queuing ..................................................................................................................... 65
`3.2.2
`Connections............................................................................................................... 69
`Communications Stack ................................................................................................. 69
`IBA Components........................................................................................................... 70
`3.4.1
`Links & Repeaters ..................................................................................................... 70
`3.4.2
`Channel Adapters...................................................................................................... 71
`3.4.3
`Switches .................................................................................................................... 72
`3.4.4
`Routers ...................................................................................................................... 73
`3.4.5
`Management Components ........................................................................................ 74
`3.4.5.1 Subnet Managers ......................................................................................... 74
`3.4.5.2 Subnet Management Agents ........................................................................ 75
`3.4.5.3 General Service Agents................................................................................ 75
`IBA Features................................................................................................................. 76
`3.5.1
`Queue Pairs............................................................................................................... 76
`3.5.2
`Types of Service ........................................................................................................ 77
`3.5.3
`Keys........................................................................................................................... 78
`
`3.5
`
`InfiniBandSM Trade Association
`
` Page 3
`
`3
`
`

`
`1234567891
`
`0
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`21
`22
`23
`24
`25
`26
`27
`28
`29
`30
`31
`32
`33
`34
`35
`36
`37
`38
`39
`40
`41
`42
`
`InfiniBandTM Architecture Release 1.0.a
`VOLUME 1 - GENERAL SPECIFICATIONS
`
`June 19, 2001
`FINAL
`
`3.5.4
`3.5.5
`3.5.6
`3.5.7
`3.5.8
`
`3.5.9
`3.5.10
`3.5.11
`
`Virtual Memory Addresses......................................................................................... 80
`Protection Domains ................................................................................................... 80
`Partitions.................................................................................................................... 81
`Virtual Lanes.............................................................................................................. 81
`Quality of Service ...................................................................................................... 82
`3.5.8.1 Service Level ................................................................................................ 83
`3.5.8.2 SL to VL mapping ......................................................................................... 83
`3.5.8.3 Partitions....................................................................................................... 83
`Injection Rate Control ................................................................................................ 83
`Addressing................................................................................................................. 84
`Multicast .................................................................................................................... 86
`3.5.11.1Multicast Example......................................................................................... 87
`3.5.11.2Group Management...................................................................................... 88
`3.5.11.3Multicast Prune............................................................................................. 91
`Verbs ......................................................................................................................... 91
`3.5.12
`Channel & Memory Semantics ..................................................................................... 91
`3.6.1
`Communication Interface........................................................................................... 92
`3.6.2
`IBA Transport Services.............................................................................................. 94
`IBA Layered Architecture.............................................................................................. 98
`3.7.1
`Physical Layer ........................................................................................................... 98
`3.7.2
`Link Layer .................................................................................................................. 99
`3.7.3
`Network Layer ......................................................................................................... 100
`3.7.4
`Transport Layer ....................................................................................................... 101
`3.7.5
`Upper Layer Protocols............................................................................................. 103
`3.7.5.1 Subnet Management .................................................................................. 103
`3.7.5.2 General Services ........................................................................................ 104
`IBA Transaction Flow.................................................................................................. 105
`IBA Management Infrastructure.................................................................................. 106
`3.9.1
`Management Interfaces............................................................................................110
`3.9.2
`Subnet Management Interface .................................................................................110
`3.9.2.1 Fabric Initialization.......................................................................................110
`3.9.2.2 Gets & Sets..................................................................................................110
`3.9.2.3 Traps and Notices........................................................................................111
`3.9.2.4 Actions.........................................................................................................111
`3.9.2.5 Directed Routes...........................................................................................111
`General Service Interface.........................................................................................112
`3.9.3.1 Management Datagrams .............................................................................112
`3.9.3.2 Redirection...................................................................................................112
`
`3.6
`
`3.7
`
`3.8
`3.9
`
`3.9.3
`
`InfiniBandSM Trade Association
`
` Page 4
`
`4
`
`

`
`1234567891
`
`0
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`21
`22
`23
`24
`25
`26
`27
`28
`29
`30
`31
`32
`33
`34
`35
`36
`37
`38
`39
`40
`41
`42
`
`InfiniBandTM Architecture Release 1.0.a
`VOLUME 1 - GENERAL SPECIFICATIONS
`
`June 19, 2001
`FINAL
`
`3.10
`I/O Operation ...............................................................................................................112
`Chapter 4:
`Addressing ...................................................................................... 115
`4.1
`Terminology And Concepts..........................................................................................116
`4.1.1
`GID Usage and Properties .......................................................................................117
`4.1.2
`Channel Adapter, Switch, and Router Addressing Rules ........................................ 121
`4.1.3
`Local Identifiers ....................................................................................................... 121
`Chapter 5:
`Data Packet Format
`........................................................................ 124
`5.1
`Packet Types .............................................................................................................. 124
`5.2
`Data Packet Format.................................................................................................... 125
`5.2.1
`Local Route Header (LRH) - 8 Bytes....................................................................... 128
`5.2.2
`Global Route Header (GRH) - 40 Bytes .................................................................. 128
`5.2.3
`Base Transport Header (BTH) - 12 Bytes................................................................ 129
`5.2.4
`Reliable Datagram Extended Transport Header (RDETH) - 4 Bytes....................... 130
`5.2.5
` Datagram Extended Transport Header (DETH) - 8 Bytes ...................................... 131
`5.2.6
`RDMA Extended Transport Header (RETH) - 16 Bytes .......................................... 131
`5.2.7
`Atomic Extended Transport Header (AtomicETH) - 28 Bytes.................................. 132
`5.2.8
`ACK Extended Transport Header (AETH) - 4 Bytes................................................ 133
`5.2.9
`Atomic ACK Extended Transport Header (AtomicAckETH) - 8 Bytes ..................... 133
`5.2.10
`Immediate Data Extended Transport Header (ImmDt) - 4 Bytes............................. 134
`5.2.11
`Payload.................................................................................................................... 134
`5.2.12
`Invariant CRC .......................................................................................................... 135
`5.2.13
`Variant CRC............................................................................................................. 135
`5.3
`Raw Packet Format .................................................................................................... 135
`5.4
`Packet Examples........................................................................................................ 136
`Chapter 6:
`Physical Layer Interface
`................................................................ 137
`6.1
`Overview..................................................................................................................... 137
`6.2
`Services provided by the Physical Layer. ................................................................... 137
`6.3
`Interface between physical and Link Layers............................................................... 138
`6.3.1
`Interface between physical receive and link receive................................................ 138
`6.3.1.1 Phy_link - Physical Link Status ................................................................... 138
`6.3.1.2 L_Init_Train - Link Initiate Retraining .......................................................... 138
`6.3.1.3 rcv_stream - Receive Stream ..................................................................... 139
`Interface between physical Transmit and link Transmit........................................... 139
`6.3.2.1 Xmit_stream - Transmit Stream .................................................................. 139
`6.3.2.2 Xmit_Ready - Physical Transmitter Ready ................................................. 139
`Link Layer
`Chapter 7:
`....................................................................................... 141
`7.1
`Overview..................................................................................................................... 141
`7.1.1
`State Machine Conventions..................................................................................... 141
`
`6.3.2
`
`InfiniBandSM Trade Association
`
` Page 5
`
`5
`
`

`
`1234567891
`
`0
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`21
`22
`23
`24
`25
`26
`27
`28
`29
`30
`31
`32
`33
`34
`35
`36
`37
`38
`39
`40
`41
`42
`
`InfiniBandTM Architecture Release 1.0.a
`VOLUME 1 - GENERAL SPECIFICATIONS
`
`June 19, 2001
`FINAL
`
`7.2
`
`7.3
`7.4
`7.5
`7.6
`
`7.7
`
`7.8
`
`Link States .................................................................................................................. 142
`7.2.1
`LinkDown State........................................................................................................ 142
`7.2.2
`LinkInitialize State.................................................................................................... 142
`7.2.3
`LinkArm State .......................................................................................................... 142
`7.2.4
`LinkActive State ....................................................................................................... 143
`7.2.5
`LinkActDefer State................................................................................................... 143
`7.2.6
`Management State Change Commands.................................................................. 143
`7.2.7
`State Machine Terms ............................................................................................... 143
`Packet Receiver States............................................................................................... 146
`Data Packet Check..................................................................................................... 148
`Link Packet Check...................................................................................................... 151
`Virtual Lanes Mechanisms.......................................................................................... 153
`7.6.1
`VL identification ....................................................................................................... 154
`7.6.2
`Number of VLs supported........................................................................................ 155
`7.6.3
`Special VLs.............................................................................................................. 155
`7.6.4
`Buffering and Flow Control For Data VLs ................................................................ 156
`7.6.5
`Service Level ........................................................................................................... 158
`7.6.6
`VL Mapping Within a Subnet ................................................................................... 159
`7.6.7
`Initialization and Configuration ................................................................................ 161
`7.6.8
`VL Scheduling and Flow Control For VL15 and Flow Control Packets.................... 161
`7.6.9
`VL Arbitration and Prioritization ............................................................................... 161
`7.6.9.1 VL Arbitration When Only One Data VL Is Implemented............................ 162
`7.6.9.2 VL Arbitration When Multiple Data VL s Are Implemented ......................... 162
`Local Route Header.................................................................................................... 165
`7.7.1
`Virtual Lane (VL) - 4 bits .......................................................................................... 166
`7.7.2
`Link Version (LVer) - 4 bits....................................................................................... 166
`7.7.3
`Service Level (SL) - 4 bits........................................................................................ 166
`7.7.4
`Reserve - 2 bits........................................................................................................ 167
`7.7.5
`Link Next Header (LNH) - 2 bits............................................................................... 167
`7.7.6
`Destination Local Identifier (DLID) - 16 bits............................................................. 167
`7.7.7
`Reserve - 5 bits........................................................................................................ 167
`7.7.8
`Packet Length (PktLen) - 11 bits.............................................................................. 167
`7.7.9
`Source Local Identifier (SLID) - 16 bits.................................................................... 168
`CRCs .......................................................................................................................... 168
`7.8.1
`Invariant CRC (ICRC) - 4 Bytes............................................................................... 168
`7.8.2
`Variant CRC (VCRC) - 2 Bytes ................................................................................ 170
`7.8.3
`Link Packet CRC (LPCRC) - 2 Bytes....................................................................... 171
`7.8.4
`CRC Calculation Samples ....................................................................................... 171
`7.8.4.1 ICRC Generator.......................................................................................... 171
`7.8.4.2 VCRC Generator ........................................................................................ 172
`7.8.4.3 Sample Packets.......................................................................................... 173
`
`InfiniBandSM Trade Association
`
` Page 6
`
`6
`
`

`
`1234567891
`
`0
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`21
`22
`23
`24
`25
`26
`27
`28
`29
`30
`31
`32
`33
`34
`35
`36
`37
`38
`39
`40
`41
`42
`
`InfiniBandTM Architecture Release 1.0.a
`VOLUME 1 - GENERAL SPECIFICATIONS
`
`June 19, 2001
`FINAL
`
`7.9
`
`7.10
`
`Flow Control ............................................................................................................... 182
`7.9.1
`Introduction.............................................................................................................. 182
`7.9.2
`Flow Control Blocks................................................................................................. 183
`7.9.3
`Relationship to Virtual Lanes................................................................................... 183
`7.9.4
`Flow Control Packet ................................................................................................ 183
`7.9.4.1 Flow Control Packet Fields......................................................................... 183
`7.9.4.2 Calculation of FCTBS ................................................................................. 184
`7.9.4.3 Calculation of FCCL.................................................................................... 184
`7.9.4.4 Transmission of Packets............................................................................. 185
`IBA and Raw Packet Multicast ................................................................................... 186
`7.10.1
`Overview.................................................................................................................. 186
`7.10.2
`IBA Unreliable Multicast Operational Rules............................................................. 187
`7.10.3
`Raw Packet Multicast .............................................................................................. 190
`7.10.3.1Raw Multicast Operational Rules................................................................ 190
`Group Management................................................................................................. 192
`7.10.4
`Subnet Multipathing.................................................................................................... 192
`7.11.1
`Multipathing Requirements on end node ................................................................. 192
`Error detection and handling ...................................................................................... 192
`7.12.1
`Error Detection ........................................................................................................ 192
`7.12.2
`Error Recovery Procedures ..................................................................................... 194
`7.12.3
`Error Notification...................................................................................................... 194
`Chapter 8:
`Network Layer
`................................................................................. 195
`8.1
`Overview..................................................................................................................... 195
`8.2
`Packet Routing ........................................................................................................... 195
`8.2.1
` Overview................................................................................................................. 195
`8.2.2
`Global Fabric Characteristics................................................................................... 196
`8.2.2.1 Inheritance of Subnet Requirements .......................................................... 196
`8.2.2.2 Packet Errors and Error Detection.............................................................. 196
`8.2.2.3 Service Levels ............................................................................................ 196
`Support for Multiple Global Paths............................................................................ 196
`8.2.3
`Global Multicast ....................................................................................................... 198
`8.2.4
`Global Route Header.................................................................................................. 198
`8.3.1
`IP Version (IPVer) - 4 bits ........................................................................................ 198
`8.3.2
`Traffic Class (TClass) - 8 bits................................................................................... 198
`8.3.3
`Flow Label (FlowLabel) - 20 bits.............................................................................. 199
`8.3.4
`Payload Length (PayLen) - 16 bits .......................................................................... 199
`8.3.5
`Next Header (NxtHdr) - 8 bits .................................................................................. 199
`8.3.6
`Hop Limit (HopLmt) - 8 bits...................................................................................... 199
`8.3.7
`Source Global Identifier (SGID) - 128 bits ............................................................... 199
`8.3.8
`Destination Global Identifier (DGID) - 128 bits ........................................................ 199
`
`7.11
`
`7.12
`
`8.3
`
`InfiniBandSM Trade Association
`
` Page 7
`
`7
`
`

`
`1234567891
`
`0
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`21
`22
`23
`24
`25
`26
`27
`28
`29
`30
`31
`32
`33
`34
`35
`36
`37
`38
`39
`40
`41
`42
`
`InfiniBandTM Architecture Release 1.0.a
`VOLUME 1 - GENERAL SPECIFICATIONS
`
`June 19, 2001
`FINAL
`
`8.4
`
`9.3
`
`Global Route Header Usage ...................................................................................... 199
`8.4.1
`Global Route Header Generation ............................................................................ 199
`8.4.2
`Global Route Header Modification........................................................................... 201
`8.4.3
`Global Route Header Verification ............................................................................ 202
`Chapter 9:
`Transport Layer
`.............................................................................. 203
`9.1
`Overview..................................................................................................................... 203
`9.2
`Base Transport Header .............................................................................................. 206
`9.2.1
`Operation Code (OpCode) ...................................................................................... 206
`9.2.2
`Reserved Transport Function OpCodes .................................................................. 208
`9.2.3
`Solicited Event (SE) - 1 bit....................................................................................... 209
`9.2.4
`MigReq (M) - 1 Bit ................................................................................................... 209
`9.2.5
`Pad Count (PadCnt) - 2 bits..................................................................................... 209
`9.2.6
`Transport Header Version (TVer) - 4 bits................................................................. 209
`9.2.7
`Partition Key (P_Key) - 16 bits................................................................................. 210
`9.2.8
`Destination QP (DestQP) - 24 bits........................................................................... 210
`9.2.9
`Reserve 8 (Resv8) - 8 bits....................................................................................... 210
`9.2.10
`AckReq (A) - 1 Bit.................................................................................................... 210
`9.2.11
`Reserve 7 (resv7) - 7 bits ........................................................................................ 210
`9.2.12
`Packet Sequence Number (PSN) - 24 bits .............................................................. 210
`Extended Transport Headers...................................................................................... 210
`9.3.1
`Reliable Datagram Extended Transport Header (RDETH) - 4 Bytes....................... 210
`9.3.1.1 Reserve - 8 bits............................................................................................211
`9.3.1.2 End-to-End (EE) Context - 24 bits ..............................................................211
`Datagram Extended Transport Header (DETH) - 8 Bytes ........................................211
`9.3.2.1 Q_Key - 32 bits ............................................................................................211
`9.3.2.2 Reserve - 8 bits............................................................................................211
`9.3.2.3 Source QP (SrcQP) - 24 bits .......................................................................211
`RDMA Extended Transport Header (RETH) - 16 Bytes .......................................... 212
`9.3.3.1 Virtual Address (VA) - 64 bits...................................................................... 212
`9.3.3.2 R_Key - 32 bits ........................................................................................... 212
`9.3.3.3 DMA Length (DMAlen) - 32 bits.................................................................. 213
`ATOMIC Extended Transport Header (AtomicETH) - 28 Bytes ............................... 213
`9.3.4.1 Virtual Address (VA) - 64 bits...................................................................... 213
`9.3.4.2 R_Key - 32 bits ........................................................................................... 213
`9.3.4.3 Swap (Add) Data (SwapDt) - 64 bits........................................................... 213
`9.3.4.4 Compare Data (CmpDt) - 64 bits ................................................................ 213
`ACK Extended Transport Header (AETH) - 4 Bytes................................................ 214
`9.3.5.1 Syndrome ................................................................................................... 214
`9.3.5.2 Message Sequence Number (MSN)........................................................... 214
`9.3.5.3 ATOMIC Acknowledge Extended Transport Header (AtomicAckETH) - 8 Bytes214
`9.3.5.4 Original Remote Data (OrigRemDt) - 64 bits .............................................. 214
`Immediate Extended Transport Header (ImmDt) - 4 Bytes ..................................... 215
`
`9.3.2
`
`9.3.3
`
`9.3.4
`
`9.3.5
`
`9.3.6
`
`InfiniBandSM Trade Association
`
` Page 8
`
`8
`
`

`
`1234567891
`
`0
`11
`12
`13
`14
`15
`16
`17
`18
`19
`20
`21
`22
`23
`24
`25
`26
`27
`28
`29
`30
`31
`32
`33
`34
`35
`36
`37
`38
`39
`40
`41
`42
`
`InfiniBandTM Architecture Release 1.0.a
`VOLUME 1 - GENERAL SPECIFICATIONS
`
`June 19, 2001
`FINAL
`
`9.4
`
`9.5
`9.6
`
`9.7
`
`Transport Functions.................................................................................................... 215
`9.4.1
` SEND Operation..................................................................................................... 215
`9.4.2
`RESYNC Operation................................................................................................. 219
`9.4.3
` RDMA WRITE Operation........................................................................................ 219
`9.4.4
` RDMA READ Operation ......................................................................................... 222
`9.4.5
`ATOMIC Operations ................................................................................................ 227
`9.4.5.1 Atomicity Guarantees ................................................................................. 229
`9.4.5.2 ATOMIC Acknowledgment Generation and Ordering Rules....................... 230
`9.4.5.3 Error Behavior............................................................................................. 230
`Reserved and Manufacturer Defined Transport Function OpCodes ....................... 235
`9.4.6
`Transaction Ordering .......................................................................................