(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2002/0091844 A1
`(43) Pub. Date:
`Jul. 11, 2002
`Craft et al.
`
`US 2002009 1844A1
`
`(54) NETWORK INTERFACE DEVICE THAT
`FASTPATH PROCESSES SOLICITED
`SESSION LAYER READ COMMANDS
`
`(75)
`
`(73)
`(21)
`(22)
`
`(63)
`
`Inventors: Peter K. Craft, San Franciso, CA (US);
`Clive M. Philbrick, San Jose, CA
`(US); Laurence B. Boucher, Saratoga,
`CA (US)
`Correspondence Address:
`T. Lester Wallace
`Patent Attorney
`Suite 280
`7041 Koll Center Parkway
`Pleasanton, CA 94566 (US)
`Assignee: Alacritech, Inc.
`Appl. No.:
`09/970,124
`
`Filed:
`
`Oct. 2, 2001
`Related U.S. Application Data
`Continuation of application No. 09/067,544, filed on
`Apr. 27, 1998, now Pat. No. 6,226,680. Continuation
`of application No. 09/141,713, filed on Aug. 28, 1998,
`now Pat. No. 6,389,479. Continuation of application
`No. 09/384,792, filed on Aug. 27, 1999. Continuation
`of application No. 09/416,925, filed on Oct. 13, 1999.
`Continuation of application No. 09/439,603, filed on
`Nov. 12, 1999, now Pat. No. 6,247,060. Continuation
`of application No. 09/464,283, filed on Dec. 15, 1999.
`Continuation of application No. 09/514,425, filed on
`Feb. 28, 2000. Continuation of application No.
`09/675,484, filed on Sep. 29, 2000. Continuation of
`application No. 09/675,700, filed on Sep. 29, 2000.
`Continuation of application No. 09/692,561, filed on
`Oct. 18, 2000. Continuation of application No.
`09/748,936, filed on Dec. 26, 2000, now Pat. No.
`
`6,334,153. Continuation of application No. 09/789,
`366, filed on Feb. 20, 2001. Continuation of applica
`tion No. 09/801,488, filed on Mar. 7, 2001. Continu
`ation of application No. 09/802,551, filed on Mar. 9,
`2001. Continuation of application No. 09/802,426,
`filed on Mar. 9, 2001. Continuation of application No.
`09/802,550, filed on Mar. 9, 2001. Continuation of
`application No. 09/804,553, filed on Mar. 12, 2001,
`now Pat. No. 6,393,487. Continuation of application
`No. 09/855,979, filed on May 14, 2001. Non-provi
`sional of provisional application No. 60/061,809,
`filed on Oct. 14, 1997. Non-provisional of provisional
`application No. 60/098,296, filed on Aug. 27, 1998.
`
`Publication Classification
`
`(51) Int. Cl." .......................... G06F 15/16; G06F 13/10;
`HO4L 12/66
`(52) U.S. Cl. ......................... 709/230; 709/250; 370/463;
`709/321
`
`(57)
`
`ABSTRACT
`
`A network interface device connected to a host provides
`hardware and processing mechanisms for accelerating data
`transferS between the host and a network. Some data trans
`fers are processed using a dedicated fast-path whereby the
`protocol Stack of the host performs no network layer or
`transport layer processing. Other data transfers are, however,
`handled in a slow-path by the host protocol Stack. In one
`embodiment, the host protocol stack has an ISCSI layer, but
`a response to a Solicited ISCSI read request command is
`nevertheless processed by the network interface device in
`fast-path. In another embodiment, an initial portion of a
`response to a Solicited command is handled using the
`dedicated fast-path and then after an error condidtion occurs
`a Subsequent portion of the response is handled using the the
`Slow-path. The interface device uses a command Status
`message to communicate Status to the host.
`
`RECEIVEFILEREQUEST FROM NETWORK, PROCESS REQLES T
`WITHSACKDIRECT TO FIESYSEM
`S.
`200
`IOCATEREQUESTEDFILEONHOST FILE CACHE, NICFELL
`CACTTE, HOSTSIORAGE UNIF ORINICSTORAGE UNIT
`S.
`202
`
`206
`-
`REAT) FILEBLOCKS
`TO NIFILE AChE
`
`204
`-
`READFILE BLOCKS /
`OHOSELECACIIE
`20
`
`22
`{
`
`CREATENETWORK
`>
`<FAST PATH
`S NO FRAMES FROM FILE
`BLOCKS IN HOST BY
`CPURNNING STACK
`
`NO
`
`Y
`
`CREATE HEADERSBY
`INC. FROM CCB, DMA
`DATA TONICWEACCB,
`PREPEND HEADERSO
`BATA TO CREATE
`NETWORK FRAMES
`
`CREATEEADTERS
`BY PRINNN:
`STACK, INSTRUC
`INTO ASSEMBE
`FRAMES FROM
`HEAERSAN FILE
`BIOCKS INNIC
`
`
`
`
`
`
`
`
`
`
`
`
`
`DWA
`FRAMES C.
`NLC
`
`
`
`CREATE HEAERS
`BY INICFRoycoB.
`: REPENIEADERS
`TODATA TO CREATE
`NETWORK FRAMES
`
`SEN FRAMES ONTONETWORK
`
`Alacritech, Ex. 2022.001
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002. Sheet 1 of 25
`
`US 2002/0091844 A1
`
`
`
`
`
`PROTOCOL
`STACK38
`
`-
`
`HOST MEMORY 33
`MEMORY FILE SYSTEM
`MGR 27
`23
`CACHEl
`| MGR 26
`| HOST FILE
`
`- - - - -
`
`- - - - -
`
`HOST
`STORAGE
`UNIT 66.
`
`62
`
`HOST I/O
`CTRL 64
`
`HOST20
`
`CPU 30
`
`BRIGE
`
`is
`
`PROCESSOR
`
`BREE
`
`DMA
`
`INIC I/O
`
`
`
`SEQUENCERS 52
`
`INIC 22
`
`60
`
`a / Yi
`
`58 - || |
`
`|
`
`INIC MEMORY 46
`
`- - - - - - - -
`
`|FRAME BUFFERS 77
`CCB |
`\, y
`- us .
`ICACHE
`74 ||NICFILE CACHE 80
`- - - - - - - -
`
`- - - - - - - -
`
`75
`
`LAN/WAN 25
`
`INTERNET 28
`
`INIC STORAGE
`UNIT 70
`
`FIG. 1
`
`Alacritech, Ex. 2022.002
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 2 of 25
`
`US 2002/0091844 A1
`
`HOST 20
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`HOST
`|
`| HOST FILE
`PROTOCOL
`STACK38 MEMORY CACHE 24
`
`
`
`
`
`FILE SYSTEM
`23
`
`81
`
`
`
`HOST STORAGE UNIT
`66
`
`INIC 22
`
`t
`
`- - - - -
`
`PROCESSOR
`44
`
`SEQUENCERS
`52
`
`
`
`90
`
`
`
`LAN/WAN 25
`
`INIC sToGE UNIT
`
`FIG. 2
`
`Alacritech, Ex. 2022.003
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002. Sheet 3 of 25
`
`US 2002/0091844 A1
`
`RECEIVE PACKET FROM NETWORK BY INIC . Y OO
`
`VALIDATE PACKET, SUMMARIZE HEADERS |- 102
`
`STORE PACKET AND SUMMARY
`IN BUFFERS
`
`104
`108
`
`FAST PATH
`CANDIDATE
`
`
`
`NO
`
`SEND PACKET TO
`STACK FOR SLOW
`PATH PROCESSING
`
`106
`
`
`
`
`
`
`
`SEND PACKET TO
`STACK FOR SLOW
`PATH PROCESSING
`
`FLUSH CCB AND
`PACKET TO STACK
`FOR SLOW-PATH
`PROCESSING
`
`120
`
`
`
`CREATE CCB FOR
`MESSAGE
`
`
`
`DETERMINE
`DESTINATION IN
`FILE CACHE,
`STORE LIST OF NO
`DESTINATION
`ADDRESSES WITH
`CCB IN INIC
`
`YES
`
`122
`
`125
`
`SEND DATA TO
`DESTINATION
`VIA FAST-PATH
`
`FIG. 3
`
`115
`
`Alacritech, Ex. 2022.004
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 4 of 25
`
`US 2002/009 1844 A1
`
`RECEIVE FILEREQUEST FROM NETWORK, PROCESS REQUEST
`WITHSTACK, DIRECT TO FILE SYSTEM
`
`200
`LOCATE REQUESTED FILE ON HOST FILE CACHE, INICFILE
`CACHE, HOST STORAGE UNIT OR INIC STORAGE UNIT
`
`204
`
`206
`
`READ FILE BLOCKS
`TO HOST FILE CACHE
`
`212
`
`READ FILE BLOCKS
`TO INIC FILE CACHE
`
`202
`
`210
`-
`
`
`
`
`
`CREATE NETWORK
`NO FRAMES FROM FILE
`BLOCKS IN HOST BY
`CPU RUNNING STACK
`
`YES
`
`218
`
`CREATE HEADERS BY
`INIC FROM CCB, DMA
`DATA TO INIC VIA CCB,
`PREPEND HEADERS TO
`DATA TO CREATE
`NETWORK FRAMES
`
`CREATE HEADERS
`BY CPU RUNNING
`STACK, INSTRUCT
`INIC TO ASSEMBLE
`FRAMES FROM
`HEADERS AND FILE
`BLOCKS IN INIC
`
`
`
`
`
`
`
`YES
`
`225
`
`214
`
`DMA
`FRAMES TO
`INIC
`
`
`
`CREATE HEADERS
`BY INIC FROM CCB,
`PREPEND HEADERS
`TODATA TO CREATE
`NETWORK FRAMES
`
`
`
`
`
`
`
`
`
`SEND FRAMES ONTONETWORK
`
`FIG. 4
`
`Alacritech, Ex. 2022.005
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 5 of 25
`
`US 2002/0091844 A1
`
`
`
`FIRST
`NETWORK
`25
`
`FIRST INC
`STORAGE
`UNIT 66.
`
`NTH
`NETWORK
`313
`
`
`
`
`
`
`
`HOST
`STORAGE
`UNIT 66.
`
`N-1 - 40
`
`SESONR
`U.
`
`SECOND ENIC
`STORAGE
`UNIT 308
`
`NTH INIC
`STORAGE
`UNIT 315
`
`NTH
`INIC 310
`--
`
`SECOND
`NETWORK
`305
`
`FIG. 5
`
`450
`S
`
`1. SAN 418
`
`
`
`
`
`FIRST
`ETHER-SCSI
`ADAPTER
`452.
`
`SECOND
`ETHER-SCSI
`ADAPTER
`454.
`
`THIRD
`ETHER-SCSI
`ADAPTER
`456
`
`INIC 400
`
`
`
`
`
`458
`
`459
`
`460
`
`FIRST
`STORAGE
`UNIT 462
`
`
`
`SECOND
`STORAGE
`UNIT 464
`
`THIRD
`STORAGE
`UNIT 466
`
`FIG. 7
`
`Alacritech, Ex. 2022.006
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 6 of 25
`
`US 2002/0091844 A1
`
`HOST
`MEMORY
`33
`
`METADATA
`CACHE 26
`
`PROTOCOL
`STACK 38
`
`HOST FILE
`CACHE 24
`
`
`
`
`
`HOST
`STORAGE
`UNIT 66
`
`HOST 20
`
`CPU 30
`
`35 BRPGE
`--
`
`HOST I/O
`CTRL 64
`
`
`
`
`
`PROCESSOR408
`
`BRIDGE
`406
`
`404
`
`INIC 400
`
`FIFF
`EEE
`
`424
`
`422 is
`
`INIC MEMORY 410
`
`FRAME BUFFERS 430
`
`INICFILE CACHE 433
`
`FIRST LAN
`414
`
`SECOND LAN
`416
`
`
`
`
`
`
`
`FIRST SAN
`418
`
`SECOND SAN
`420
`
`FIG. 6
`
`Alacritech, Ex. 2022.007
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 7 of 25
`
`US 2002/0091844 A1
`
`
`
`
`
`
`
`PROCESSOR 480
`
`FIRST
`ETHER-SCSI
`ADAPTER
`452
`
`
`
`ADAPTER MEMORY 482
`
`SEQUENCERS475
`
`
`
`
`
`MAC 47 2 - SCSI
`CTRL 488
`PHY 470
`
`FRAME BUFFERS 484
`
`FILE CACHE 486
`
`
`
`
`
`FIRST
`STORAGE
`UNIT 462
`
`FIG. 8
`
`HOST
`STORAGE
`UNIT 66.
`
`
`
`NTH
`INIC 490
`
`
`
`
`
`NTHSAN
`492
`
`
`
`FIRST LAN
`414
`
`
`
`
`
`SECOND LAN
`416
`
`FIRST
`INIC 400
`
`
`
`
`
`
`
`FIRST SAN
`418
`
`SECOND SAN
`420
`
`FIG. 9
`
`Alacritech, Ex. 2022.008
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 8 of 25
`
`US 2002/0091844 A1
`
`MEDIA ACCESS
`CONTROLLER
`
`60
`
`
`
`
`
`
`
`ASSEMBLY
`REGISTER
`
`
`
`
`
`
`
`
`
`
`
`FLY BY
`SEQUENCER
`
`PACKET
`CONTROL
`SEQUENCER
`
`MULTIPLEXER
`
`508
`
`SRAM
`
`
`
`DRAM CONTROL
`
`
`
`QUEUE
`MANAGER
`
`FIG. 10
`
`Alacritech, Ex. 2022.009
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 9 of 25
`
`US 2002/0091844 A1
`
`
`
`510
`
`
`
`502
`
`
`
`
`
`
`
`
`
`
`
`
`
`ASSEMBLY
`REGISTER
`
`500
`
`
`
`
`
`
`
`PACKET
`CONTROL
`SEQUENCER
`
`MAC
`SEQUENCER
`
`NETWORK
`SEQUENCER
`
`TRANSPORT
`SEQUENCER
`
`SESSION
`SEQUENCER
`
`505
`
`MULTIPLEXOR
`
`FIG. 11
`
`Alacritech, Ex. 2022.010
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 10 of 25 US 2002/0091844A1
`
`UPPER LAYER
`
`UPPER LAYER
`INTERFACE
`
`TRANSPORT
`
`
`
`
`
`
`
`
`
`NETWORK
`
`DATA LINK
`
`38
`
`570
`
`568
`
`566
`
`564
`
`562
`
`
`
`560
`
`INIC DEVICE DRIVER
`
`FIG. 12
`
`38 Y
`
`
`
`TDI USERS
`
`575
`
`572/N
`
`TDIFILTER DRIVER
`& UPPERLAYER INTERFACE
`
`590
`
`548
`
`580
`
`
`
`
`
`570
`
`INIC DEVICE DRIVER
`
`FIG. 13
`
`Alacritech, Ex. 2022.011
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 11 of 25 US 2002/0091844 A1
`
`
`
`2
`VC
`
`H
`
`a 2
`5
`?t
`r
`D. O St.
`269
`3.
`ass M
`
`O ?t
`> 5
`t
`Z2
`CM)
`
`r
`O
`3
`3 a
`Z3
`M)
`M
`<
`Z.
`
`- - - - - s - a- - -a- - - - - am - am - a- - - - - - are - are
`
`of
`NC & N
`
`N
`
`ON
`
`ass: & S
`VO
`O
`Han
`- 2 –
`
`M
`
`-
`H t
`t
`
`C O
`a
`O a
`<
`H
`>
`N
`
`/
`
`c CN
`
`3
`| A
`?t
`| D.
`Y.
`|
`in
`M
`
`------------- - M - - - - - - - - - - O - m -
`
`;
`
`:
`
`S a
`S c
`
`- - - - - - - - - - - - - m - m -- - M - O - O - - - - - t
`
`N.
`\d
`
`vC
`
`r
`Ym
`O
`t
`
`S.
`
`No
`
`Md
`
`\d
`
`| ||
`
`CN
`
`CN S
`.
`Z
`c a Sol
`fr
`C A.
`O s
`O || 7 |
`-
`Z.
`
`- T / I
`
`wn
`
`OO
`
`3
`
`.
`
`Wo
`
`S
`S
`Z.
`m
`
`Md
`\d O
`A. <
`>
`
`rud
`
`Alacritech, Ex. 2022.012
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 12 0f 25
`
`US 2002/0091844 A1
`
`Mm>mwm
`
`QEO<HH<
`
`mUEOHm
`
`mmuhp7mw
`
`03.E:—u—gE:_a_.vammmmg0%“yam/tamu_nu_n_
`omQEoED<“HII"892292
`
`mo<§mbéa8—fl_I_“go/#5257:
`1:u"8BE_coofl_mnc_Nwm
`@034—_u—wow32u_Ill_@an_2%msma;_wadmg
`
`
`
`59555955
`
`
`
`whoNww
`
`
`
`$5320<mm2<o
`
`«mm
`
`Alacritech, Ex. 2022.013
`
`__fl
`
`Alacritech, Ex. 2022.013
`
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 13 of 25 US 2002/0091844A1
`
`
`
`EXTERNAL
`MEMORY
`CTRL
`
`Alacritech, Ex. 2022.014
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 14 of 25 US 2002/0091844A1
`
`w CONTROLS FOR FIRST REGISTER SET
`
`/
`8004,
`
`790
`
`805
`
`833
`
`FIRST REGISTER SET
`
`
`
`
`
`
`
`INSTRUCTION DECODER
`AND
`OPERAND MULTIPLEXER
`
`SECOND REGISTER SET
`
`ARITHMETIC LOGIC UNI
`
`Alacritech, Ex. 2022.015
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 15 of 25
`
`US 2002/0091844 A1
`
`08
`
`S [8 ] / [[8 ] r ()[8808
`
`Ç98
`
`
`CIVOTICIVOTI HOLEH{DVIH
`TRIJLOTRILO | TRILO{{}}COGICI
`
`| || |_|
`
`YHCICIW NICI
`
`
`
`NICI (HCICIW
`
`(C)C)
`SO3HYI | SOO§§XLOvyCI
`
`{DV TIH I OTVHH{{IIHYHCICIV
`
`
`
`@HTIH WVRI
`
`
`
`Y[HELSI OFTRI
`
`098 | 2878 || „SZ8
`
`
`
`
`
`
`
`CIVOTI ? CIVOT | WVRIS
`TRILO | TRILO | TRILO
`)– – – – – | \
`\ 008
`
`L- - - - - - -
`
`--N - - - - - - - - - - -a- -
`
`Z6
`
`Alacritech, Ex. 2022.016
`
`

`

`Patent Application Publication
`
`US 2002/0091844 A1
`
`
`
`NICI R?C?CIV
`
`|O
`
`Y?ONI
`
`898
`
`HOLVYHOS
`
`WVRIS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Alacritech, Ex. 2022.017
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 17 0f 25
`
`US 2002/0091844 A1
`
`mow
`
`
`15:0mGZEUNmDANG
`
`Q<OAMO<HmEEO
`
`N32
`
`
`
`wbmmm20m8ng95m$3ammobi3%ES
`
`mamaMufimomdivEmam/No4mm5%NW8050x5
`
`
`of.UE
`
`L..__..__.__.____________.______
`
`IIII I JI
`
`\ \,
`
`com
`
`Alacritech, Ex. 2022.018
`
`Alacritech, Ex. 2022.018
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 18 of 25 US 2002/0091844A1
`
`
`
`1022
`
`1033
`
`- 1000 460
`
`1045
`S->
`
`1018 058 1056 1054
`
`1052
`
`1050
`ND
`
`104 1072 1070 1068
`
`1066
`
`1055
`N->
`
`1076 1084 1082 1080
`
`1078
`
`1060
`
`N->
`
`1096
`
`1088
`
`2
`1004 ' 1090
`FIG. 20
`
`s
`i
`L
`
`1086
`
`1040
`
`Alacritech, Ex. 2022.019
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 19 of 25 US 2002/0091844A1
`
`1 102
`
`1110
`
`PrOc
`Req
`
`Seq
`Seq
`Seq
`Seq
`Req/ VReq/ VReq/ VReq
`1108
`1 104
`
`1144
`
`1115
`
`1146
`
`ARBITER
`
`
`
`REGISTERA
`
`1128
`
`QALU
`
`1122
`
`Y
`- - - - - - - - - - - - - - - - - - - - - - -
`Body Body
`| | | Sram Sram
`ay
`y
`Re
`Addr Write Read
`C
`Req || Req
`
`Q
`Q
`Out
`In
`Full
`Empt
`| Empty Full RDyRDy
`
`
`
`1055
`
`1060 1050 1045
`
`1030 1033 1035 1038
`FIG 21
`
`1140
`
`Alacritech, Ex. 2022.020
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 20 of 25 US 2002/0091844 A1
`
`LRU
`1200
`Y- RO R1 || R2
`|
`|
`| 7 |
`
`. .
`
`MRU
`R13 R14 R5
`13 || 4 || 0
`
`.
`
`.
`
`R7 R8 R9
`. 2 || 2 || 0 |
`FIG.22A
`
`LRU
`1200
`NR0 R1 || R2
`|| 7 || 5 |
`
`. .
`
`MRU
`R13 R14 R5
`|| 4 || 6 || 9
`
`"
`
`'
`
`R7 R8 R9
`12 || 0 || 3 |
`FIG. 22B
`
`LRU
`1200
`Y RO R1 R2
`1
`|| 7 ||
`5 ||
`
`MRU
`R13 R14 R15
`6
`9
`10
`
`R7 R8 R9
`12 || 3 || 8
`FIG. 22C
`
`LRU
`1200
`Y RO R1 || R2
`8
`1
`7
`
`O
`
`R7 R8 R9
`2
`12
`3
`FIG. 22D
`
`MRU
`R13 R14 R15
`6
`9
`10
`
`Alacritech, Ex. 2022.021
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002. Sheet 21 of 25
`
`US 2002/0091844 A1
`
`
`
`RIOSS@HOORICI
`
`
`
`
`
`SLIT) ORHIO OIOOTI
`
`Alacritech, Ex. 2022.022
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 22 of 25 US 2002/0091844A1
`
`NETWORK
`2105
`2101
`PHYSICAL
`LAYER
`INTERF.
`
`INIC 22
`-
`
`712
`
`s
`
`ACCESS
`CTRL
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`SAM
`32 BYTES
`32 BYTES
`
`;
`
`:
`
`I
`
`|
`
`PROC
`470
`TX 1 RX
`2231,2232):
`475
`:
`756
`
`i
`
`HOST 20
`------
`|
`|
`|CPU 30
`207 --------
`
`
`
`757
`HOST
`STORAGE
`UNIT 66.
`
`BUFFER
`2114
`
`Alacritech, Ex. 2022.023
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 23 of 25 US 2002/0091844 A1
`
`QUEUE
`MANAGER
`
`2210 2209
`2105
`
`QUEUE
`MANAGER
`INTERFACE
`
`PACKET
`PROCESSING SUMMARY
`SEQUENCER
`2224
`2204
`
`
`
`2228
`
`DMA
`CONTROL
`2206
`
`2230
`
`|
`
`|
`
`MAC
`
`2213 224 2227
`
`225
`8
`-------------------
`DATA
`2200
`SYNC
`BUFFER
`
`
`
`E
`
`-- - - - -
`
`- - - - - - - - - - -
`
`
`
`
`
`
`
`ASSEMBLY
`REGISTER
`
`STATUS
`PROTOCOL
`ANALYZER
`2203
`
`si--- 2225
`SRAM
`CONTROLLER
`
`|
`«- 2226
`DRAM
`CONTROLLER
`
`2214
`
`STATUS
`
`DATA
`
`2223
`
`FIG. 25
`
`Alacritech, Ex. 2022.024
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 24 of 25 US 2002/0091844 A1
`
`INIC 22
`
`2306 2307
`
`
`
`? TCP/IPSESDATA
`
`--
`
`HOST 20
`DESTINATION
`IN HOST
`FILE CACHE
`2311
`
`-
`
`MULTI-
`PACKET
`MESSAGE
`
`2315
`
`
`
`--
`TCP/IP DATA
`2313
`
`2304
`
`TCP/IP DATA
`
`FIG. 26
`
`Alacritech, Ex. 2022.025
`
`

`

`Patent Application Publication
`
`Jul. 11, 2002 Sheet 25 of 25 US 2002/0091844 A1
`
`SCS LAYER 2412
`
`coyputer 2401
`HOST COMPUTER 2407 TTT
`f/
`247
`SCSI SMB
`cCB PROTOCOL
`Tip FAST-
`stack 241
`PATH
`DESTINATION
`|MAC
`/STORAGE
`NIDDRIVER
`2413
`?
`2410
`CCBM2418
`L 2409 ---L--
`ISCSI
`READ
`FC.B.S.. INTERFACE
`REQUEST-> - DEVICE
`COMMAND F (-e-e-
`
`2400
`A1
`
`ISCSI
`TARGET 2404
`-------
`NETWORK
`STORAGEDEVICE
`
`|
`
`1N
`
`- 2408
`
`i
`
`NETWORK
`
`COMMAND
`DATAPAYLOADS
`FAST-PATH TO STATUS MESSAGE
`DESTINATION
`2419
`
`SCSI READ REQUEST
`COMMAND 2402
`Nisy"
`
`TCP PACKETS WITH DATAPAYLOADS
`BACKFROM SCSARGET
`
`POINTERTO
`ORIGINAL
`SC gAND
`
`
`
`FIG. 27
`
`2419
`y
`
`PTR
`sTATUS
`RESID INDICATION
`
`COMMAND STATUS
`MESSAGE
`
`FIG. 28
`
`STATUS INCLUDING
`SCSI COMMAND SENT
`FL
`BITAND thED BIT
`
`INDICATION OF PART
`OF DESTINATION
`THAT STILL NEEDS
`TO BE FILLED
`2422
`
`Alacritech, Ex. 2022.026
`
`

`

`US 2002/009 1844 A1
`
`Jul. 11, 2002
`
`NETWORK INTERFACE DEVICE THAT
`FAST-PATH PROCESSES SOLICITED SESSION
`LAYER READ COMMANDS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`0001. The present application claims the benefit under 35
`USC S119 of U.S. patent application Ser. No. 60/061,809,
`filed Oct. 14, 1997, and U.S. patent application Ser. No.
`60/098,296, filed Aug. 27, 1998, and claims the benefit
`under 35 USC S 120 of U.S. patent application Ser. No.
`09/067,544, filed Apr. 27, 1998, U.S. patent application Ser.
`No. 09/141,713, filed Aug. 28, 1998, U.S. patent application
`Ser. No. 09/384,792, filed Aug. 27, 1999, U.S. patent appli
`cation Ser. No. 09/416,925, filed Oct. 13, 1999, U.S. patent
`application Ser. No. 09/439,603, filed Nov. 12, 1999, U.S.
`patent application Ser. No. 09/464,283, filed Dec. 15, 1999,
`U.S. patent application Ser. No. 09/514,425, filed Feb. 28,
`2000, U.S. patent application Ser. No. 09/675,484, filed Sep.
`29, 2000, U.S. patent application Ser. No. 09/675,700, filed
`Sep. 29, 2000, U.S. patent application Ser. No. 09/692,561,
`filed Oct. 18, 2000, U.S. patent application Ser. No. 09/748,
`936, filed Dec. 26, 2000, U.S. patent application Ser. No.
`09/789,366, filed Feb. 20, 2001, U.S. patent application Ser.
`No. 09/801,488, filed Mar. 7, 2001, U.S. patent application
`Ser. No. 09/802.551, filed Mar. 9, 2001, U.S. patent appli
`cation Ser. No. 09/802,426, filed Mar. 9, 2001, U.S. patent
`application Ser. No. 09/802,550, filed Mar. 9, 2001, U.S.
`patent application Ser. No. 09/804,553, filed Mar. 12, 2001,
`and the U.S. patent application Ser. No. 09/855,979, filed
`May 14, 2001, all of which are incorporated by reference
`herein.
`
`BACKGROUND
`0002. Over the past decade, advantages of and advances
`in network computing have encouraged tremendous growth
`of computer networks, which has in turn Spurred more
`advances, growth and advantages. With this growth, how
`ever, dislocations and bottlenecks have occurred in utilizing
`conventional network devices. For example, a CPU of a
`computer connected to a network may spend an increasing
`proportion of its time processing network communications,
`leaving less time available for other work. In particular,
`demands for moving file data between the network and a
`Storage unit of the computer, Such as a disk drive, have
`accelerated. Conventionally Such data is divided into pack
`ets for transportation over the network, with each packet
`encapsulated in layers of control information that are pro
`cessed one layer at a time by the CPU of the receiving
`computer. Although the speed of CPUs has constantly
`increased, this protocol processing of network messages
`Such as file transferS can consume most of the available
`processing power of the fastest commercially available
`CPU.
`0003. This situation may be even more challenging for a
`network file Server whose primary function is to Store and
`retrieve files, on its attached disk or tape drives, by trans
`ferring file data over the network. AS networks and databases
`have grown, the Volume of information Stored at Such
`ServerS has exploded, exposing limitations of Such Server
`attached Storage. In addition to the above-mentioned prob
`lems of protocol processing by the host CPU, limitations of
`parallel data channels. Such as conventional Small computer
`
`System interface (SCSI) interfaces have become apparent as
`Storage needs have increased. For example, parallel SCSI
`interfaces restrict the number of Storage devices that can be
`attached to a Server and the distance between the Storage
`devices and the Server.
`0004 As noted in the book by Tom Clark entitled
`“Designing Storage Area Networks,” (copyright 1999)
`incorporated by reference herein, one Solution to the limits
`of server-attached parallel SCSI storage devices involves
`attaching other file Servers to an existing local area network
`(LAN) in front of the network server. This network-attached
`storage (NAS) allows access to the NAS file servers from
`other Servers and clients on the network, but may not
`increase the Storage capacity dedicated to the original net
`work server. Conversely, NAS may increase the protocol
`processing required by the original network Server, Since
`that server may need to communicate with the various NAS
`file servers. In addition, each of the NAS file servers may in
`turn be Subject to the Strain of protocol processing and the
`limitations of Storage interfaces.
`0005 Storage area networking (SAN) provides another
`Solution to the growing need for file transfer and Storage
`over networks, by replacing daisy-chained SCSI Storage
`devices with a network of Storage devices connected behind
`a Server. Instead of conventional network Standards Such as
`Ethernet or Fast Ethernet, SANS deploy an emerging net
`working standard called Fibre Channel (FC). Due to its
`relatively recent introduction, however, many commercially
`available FC devices are incompatible with each other. Also,
`a FC network may dedicate bandwidth for communication
`between two points on the network, Such as a Server and a
`Storage unit, the bandwidth being wasted when the points are
`not communicating.
`0006 NAS and SAN as known today can be differenti
`ated according to the form of the data that is transferred and
`stored. NAS devices generally transfer data files to and from
`other file servers or clients, whereas device level blocks of
`data may be transferred over a SAN. For this reason, NAS
`devices conventionally include a file System for converting
`between files and blocks for storage, whereas a SAN may
`include Storage devices that do not have Such a file System.
`0007 Alternatively, NAS file servers can be attached to
`an Ethernet-based network dedicated to a Server, as part of
`an Ethernet SAN. Marc Farley further states, in the book
`“Building Storage Networks,” (copyright 2000) incorpo
`rated by reference herein, that it is possible to run Storage
`protocols over Ethernet, which may avoid Fibre Channel
`incompatibility issues. Increasing the number of Storage
`devices connected to a Server by employing a network
`topology Such as SAN, however, increases the amount of
`protocol processing that must be performed by that Server.
`AS mentioned above, Such protocol processing already
`Strains the most advanced Servers.
`0008 An example of conventional processing of a net
`work message Such as a file transfer illustrates Some of the
`processing Steps that slow network data Storage. A network
`interface card (NIC) typically provides a physical connec
`tion between a host and a network or networks, as well as
`providing media access control (MAC) functions that allow
`the host to access the network or networks. When a network
`message packet sent to the host arrives at the NIC, MAC
`layer headers for that packet are processed and the packet
`
`Alacritech, Ex. 2022.027
`
`

`

`US 2002/009 1844 A1
`
`Jul. 11, 2002
`
`undergoes cyclical redundancy checking (CRC) in the NIC.
`The packet is then sent across an input/output (I/O) bus Such
`as a peripheral component interconnect (PCI) bus to the
`host, and stored in host memory. The CPU then processes
`each of the header layers of the packet Sequentially by
`running instructions from the protocol Stack. This requires a
`trip across the host memory bus initially for Storing the
`packet and then Subsequent trips across the host memory bus
`for Sequentially processing each header layer. After all the
`header layerS for that packet have been processed, the
`payload data from the packet is grouped in a file cache with
`other similarly-processed payload packets of the message.
`The data is reassembled by the CPU according to the file
`System as file blocks for Storage on a disk or disks. After all
`the packets have been processed and the message has been
`reassembled as file blocks in the file cache, the file is sent,
`in blocks of data that may be each built from a few payload
`packets, back over the host memory bus and the I/O bus to
`host Storage for long term Storage on a disk, typically via a
`SCSI bus that is bridged to the I/O bus.
`0009. Alternatively, for storing the file on a SAN, the
`reassembled file in the file cache is sent in blocks back over
`the host memory bus and the I/O bus to an I/O controller
`configured for the SAN. For the situation in which the SAN
`is a FC network, a specialized FC controller is provided
`which can Send the file blocks to a storage device on the
`SAN according to Fibre Channel Protocol (FCP). For the
`situation in which the file is to be stored on a NAS device,
`the file may be directed or redirected to the NAS device,
`which processes the packets much as described above but
`employs the CPU, protocol stack and file system of the NAS
`device, and Stores blocks of the file on a Storage unit of the
`NAS device.
`0.010 Thus, a file that has been sent to a host from a
`network for storage on a SAN or NAS connected to the host
`typically requires two trips across an I/O bus for each
`message packet of the file. In addition, control information
`in header layers of each packet may cross the host memory
`bus repeatedly as it is temporarily Stored, processed one
`layer at a time, and then Sent back to the I/O bus. Retrieving
`Such a file from Storage on a SAN in response to a request
`from a client also conventionally requires significant pro
`cessing by the host CPU and file system.
`
`SUMMARY
`0.011) An interface device such as an intelligent network
`interface card (INIC) for a local host is disclosed that
`provides hardware and processing mechanisms for acceler
`ating data transferS between a network and a Storage unit,
`while control of the data transferS remains with the host. The
`interface device includes hardware circuitry for processing
`network packet headers, and can use a dedicated fast-path
`for data transfer between the network and the Storage unit,
`the fast-path set up by the host. The host CPU and protocol
`Stack avoid protocol processing for data transfer over the
`fast-path, releasing hostbus bandwidth from many demands
`of the network and Storage Subsystem. The Storage unit,
`which may include a redundant array of independent disks
`(RAID) or other configurations of multiple drives, may be
`connected to the interface device by a parallel channel Such
`as SCSI or by a serial channel such as Ethernet or Fibre
`Channel, and the interface device may be connected to the
`
`local host by an I/O bus such as a PCI bus. An additional
`Storage unit may be attached to the local host by a parallel
`interface Such as SCSI.
`0012. A file cache is provided on the interface device for
`Storing data that may bypass the host, with organization of
`data in the interface device file cache controlled by a file
`System on the host. With this arrangement, data transfers
`between a remote host and the Storage units can be processed
`over the interface device fast-path without the data passing
`between the interface device and the local host over the I/O
`bus. Also in contrast to conventional communication proto
`col processing, control information for fast-path data does
`not travel repeatedly over the host memory bus to be
`temporarily Stored and then processed one layer at a time by
`the host CPU. The host may thus be liberated from involve
`ment with a vast majority of data traffic for file reads or
`writes on host controlled Storage units.
`0013 Additional interface devices may be connected to
`the host via the I/O bus, with each additional interface
`device having a file cache controlled by the host file System,
`and providing additional network connections and/or being
`connected to additional Storage units. With plural interface
`devices attached to a single host, the host can control plural
`Storage networks, with a vast majority of the data flow to and
`from the host-controlled networks bypassing host protocol
`processing, travel acroSS the I/O bus, travel acroSS the host
`bus, and Storage in the host memory. In one example, Storage
`units may be connected to Such an interface device by a
`Gigabit Ethernet network, offering the speed and bandwidth
`of Fibre Channel without the drawbacks, and benefiting
`from the large installed base and compatibility of Ethernet
`based networks.
`0014.
`In some embodiments, a host computer whose
`protocol Stack includes an ISCSI layer is coupled to a
`network interface device. A solicited ISCSI read request
`command is Sent from the network interface device to a
`network Storage device and the network Storage device
`sends an ISCSI response back. The network interface device
`fast-path processes the ISCSI response Such that a data
`portion of the ISCSI response is placed into a destination
`memory on the host computer without the protocol Stack of
`the host computer doing any network layer or transport layer
`processing. In Some embodiments, there is no slow-path
`processing of ISCSI responses on the host. In other embodi
`ments, there is a Slow-path for handling ISCSI responses
`Such that the host protocol Stack does carry out network
`layer and transport layer processing on ISCSI responses in
`certain circumstances.
`0.015 Embodiments are described wherein an ISCSI read
`request command is Sent from a network interface device to
`an ISCSI target. The network interface device does fast-path
`processing on an initial part of the response from the ISCSI
`target, but then Switches to Slow-path processing Such that a
`Subsequent part of the response from the ISCSI target is
`processed by the host protocol Stack. In Some embodiments,
`the network interface device Sends a “command Status
`message” to the host computer to inform the host computer
`of a status condition associated with the ISCSI read request
`command. The Status condition may be an error condition.
`The Status portion of the “command Status message' may
`include a “command sent bit and/or a “flushed' bit. The
`“command Status message' may also indicate a part of a
`
`Alacritech, Ex. 2022.028
`
`

`

`US 2002/009 1844 A1
`
`Jul. 11, 2002
`
`storage destination in the host that will remain to be filled
`after the connection is flushed back to the host for slow-path
`processing. In this way, a single network cable extending
`from a single port on a network interface device can Simul
`taneously carry both ordinary network traffic as well as IP
`Storage traffic. The Same fast-path hardware circuitry on the
`network interface device is used to accelerate both the
`ordinary network traffic as well as the IP storage traffic. Error
`conditions and exception conditions for both the ordinary
`network traffic as well as the IP storage traffic are handled in
`Slow-path by the host protocol Stack.
`0016 Other embodiments are described. This summary
`does not purport to define the invention. The claims, and not
`this Summary, define the invention.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0017 FIG. 1 is a plan view diagram of a network storage
`System including a host computer connected to plural net
`works by an intelligent network interface card (INIC) having
`an I/O controller and file cache for a storage unit attached to
`the INIC.
`0.018
`FIG. 2 is a plan view diagram of the functioning of
`an INIC and host computer in transferring data between
`plural networks according to the present invention.
`0.019
`FIG. 3 is a flowchart depicting a sequence of steps
`involved in receiving a message packet from a network by
`the system of FIG. 1.
`0020 FIG. 4 is a flowchart depicting a sequence of Steps
`involved in transmitting a message packet to a network in
`response to a request from the network by the System of
`FIG. 1.
`FIG. 5 is a plan view diagram of a network storage
`0021
`System including a host computer connected to plural net
`WorkS and plural Storage units by plural INICS managed by
`the host computer.
`0022 FIG. 6 is a plan view diagram of a network storage
`System including a host computer connected to plural LANS
`and plural SANs by an intelligent network interface card
`(INIC) without an I/O controller.
`0023 FIG. 7 is a plan view diagram of a one of the SANs
`of FIG. 6, including Ethernet-SCSI adapters coupled
`between a network line and a Storage unit.
`0024 FIG. 8 is a plan view diagram of one of the
`Ethernet-SCSI adapters of FIG. 6.
`0.025
`FIG. 9 is a plan view diagram of a network storage
`System including a host computer connected to plural LANS
`and plural SANs by plural INICs managed by the host
`computer.
`0.026
`FIG. 10 is a diagram of hardware logic for the
`INIC embodiment shown in FIG. 1, including a packet
`control Sequencer and a fly-by Sequencer.
`0.027
`FIG. 11 is a diagram of the fly-by sequencer of
`FIG. 10 for analyzing header bytes as they are received by
`the INIC.
`0028 FIG. 12 is a diagram of the specialized host
`protocol Stack of FIG. 1 for creating and controlling a
`communication cont