111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US008805948B2
`
`(12) United States Patent
`Boucher et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,805,948 B2
`Aug. 12,2014
`
`(54)
`
`INTELLIGENT NETWORK INTERFACE
`SYSTEM AND METHOD FOR PROTOCOL
`PROCESSING
`
`(52)
`
`(71) Applicant: Alacritech, Inc., San Jose, CA (US)
`
`(72)
`
`Inventors: Laurence B. Boucher, Saratoga, CA
`(US); Stephen E. J. Blightman, San
`Jose, CA (US); Peter K. Craft, San
`Franscisco, CA (US); David A. Higgen,
`Apopka, FL (US); Clive M. Philbrick,
`San Jose, CA (US); Daryl D. Starr,
`Milpitas, CA (US)
`
`(73) Assignee: A-Tech LLC, Newark, DE (US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.c. 154(b) by 0 days.
`
`(21) Appl. No.: 14/038,297
`
`(22) Filed:
`
`Sep.26,2013
`
`(65)
`
`Prior Publication Data
`
`US 2014/0059155 Al
`
`Feb. 27, 2014
`
`(58)
`
`U.S. Cl.
`CPC .............. H04L 61125 (2013.01); H04L 691168
`(2013.01); H04L 69/08 (2013.01); H04Q
`2213/13204 (2013.01); H04L 69116 (2013.01);
`H04L 29106 (2013.01); H04L 69/22 (2013.01);
`H04L 67134 (2013.01); H04Q 2213/1332
`(2013.01); H04L 29112009 (2013.01); H04L
`691166 (2013.01); H04L 451245 (2013.01);
`H04L 671325 (2013.01); H04L 67110
`(2013.01); H04L 691163 (2013.01); H04L
`45100 (2013.01); H04L 691165 (2013.01);
`H04Q 310029 (2013.01); H04L 69/12
`(2013.01); H04L 691161 (2013.01); H04L
`61110 (2013.01); H04L 4919094 (2013.01);
`H04L 671327 (2013.01); H04L 29112018
`(2013.01); H04L 4919063 (2013.01); H04Q
`2213/13103 (2013.01); G06F SilO (2013.01);
`H04L 691162 (2013.01); H04Q 2213/13345
`(2013.01); H04L 69/32 (2013.01); H04Q
`2213/13093 (2013.01); H04L 49190 (2013.01);
`H04L 491901 (2013.01); H04Q 2213/13299
`(2013.01)
`USPC ............................ 7091212; 709/224; 370/474
`Field of Classification Search
`CPC ..... H04L 69119; H04L 691161; H04L 691163;
`H04L 691165; H04L 691168
`USPC ......... 709/200-202,212,224,230,236,238;
`370/389,392,419,463,469,479
`See application file for complete search history.
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 09/692,561, filed on
`Oct. 18, 2000, now Pat. No. 8,631,140, which is a
`continuation of application No. 091067,544, filed on
`Apr. 27, 1998, now Pat. No. 6,226,680.
`
`(60) Provisional application No. 601061,809, filed on Oct.
`14, 1997.
`
`(51)
`
`Int. Cl.
`G06F 15116
`H04L29106
`H04L29108
`H04L29112
`H04L 12156
`H04Q3100
`G06F SilO
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,366,538 A
`4,485,455 A
`4,485,460 A
`4,589,063 A
`4,700,185 A
`4,991,133 A
`5,056,058 A
`5,058,110 A
`5,097,442 A
`5,129,093 A
`5,163,131 A
`5,212,778 A
`
`12/1982 Johnson et al.
`1111984 Boone et al.
`1111984 Stambaugh
`5/1986 Shah et al.
`10/1987 Balph et al.
`2/1991 Davis et al.
`10/1991 Hirata et al.
`10/1991 Beach et al.
`3/1992 Ward et al.
`7/1992 Muramatsu et al.
`1111992 Row et al.
`5/1993 Dally et al.
`
`

`

`US 8,805,948 B2
`Page 2
`
`5,274,768 A
`5,280,477 A
`5,281,963 A
`5,289,580 A
`5,303,344 A
`5,412,782 A
`5,418,912 A
`5,448,566 A
`5,485,455 A
`5,485,460 A
`5,485,579 A
`5,506,966 A
`5,511,169 A
`5,517,668 A
`5,524,250 A
`5,535,375 A
`5,548,730 A
`5,553,241 A
`5,566,170 A
`5,574,919 A
`5,588,121 A
`5,590,328 A
`5,592,622 A
`5,596,574 A
`5,598,410 A
`5,619,650 A
`5,629,933 A
`5,633,780 A
`5,634,099 A
`5,634,127 A
`5,642,482 A
`5,664,114 A
`5,671,355 A
`5,678,060 A
`5,682,534 A
`5,684,954 A
`5,692,130 A
`5,699,317 A
`5,699,350 A
`5,701,434 A
`5,701,516 A
`5,706,514 A
`5,727,142 A
`5,742,765 A
`5,749,095 A
`5,751,715 A
`5,751,723 A
`5,752,078 A
`5,758,084 A
`5,758,089 A
`5,758,186 A
`5,758,194 A
`5,768,618 A
`5,771,349 A
`5,774,660 A
`5,778,013 A
`5,778,419 A
`5,790,804 A
`5,794,061 A
`5,799,150 A
`5,802,258 A
`5,802,580 A
`5,809,328 A
`5,809,527 A
`5,812,775 A
`5,815,646 A
`5,819,111 A
`5,828,835 A
`5,848,293 A
`5,870,394 A
`5,872,919 A
`5,878,225 A
`5,878,227 A
`5,892,903 A
`5,898,713 A
`5,913,028 A
`5,915,094 A
`5,917,828 A
`5,920,566 A
`
`1211993 Traw et al.
`111994 Trapp
`111994 Ishikawa et al.
`211994 Latif et al.
`411994 Yokoyama et al.
`511995 Hausman et al.
`511995 Christenson
`911995 Richter et al.
`111996 Dobbins et al.
`111996 Schrier et al.
`111996 Hitz et al.
`411996 Ban
`411996 Suda
`511996 Szwerinski et al.
`611996 Chesson et al.
`711996 Eshel et al.
`811996 Young et al.
`911996 Shirakhar
`1011996 Bakke et al.
`1111996 N etravali et al.
`1211996 Reddin et al.
`1211996 Seno et al.
`111997 Isfeld et al.
`111997 Perlman et al.
`111997 Stone
`411997 Bach et al.
`511997 Delp et al.
`511997 Cronin
`511997 Andrews et al.
`511997 Cloud et al.
`611997 Pardillos
`911997 Krech, Jr. et al.
`911997 Collins
`1011997 Yokoyama et al.
`1011997 Kapoor et al.
`1111997 Kaiserwerth et al.
`1111997 Shobu et al.
`1211997 Sartore et al.
`1211997 Kraslavsky
`1211997 Nakagawa
`1211997 Cheng et al.
`111998 Bonola
`311998 Chen
`411998 Wong etal.
`511998 Hagersten
`511998 Chan et al.
`511998 Vanden Heuvel et al.
`511998 Delp et al.
`511998 Silverstein et al.
`511998 Gentry et al.
`511998 Hamilton et al.
`511998 Kuzma
`611998 Erickson et al.
`611998 Picazo, Jr. et al.
`611998 Brendel et al.
`711998 Jedwab
`711998 Hansen et al.
`811998 Osborne
`811998 Hansen et al.
`811998 Hamilton et al.
`911998 Chen
`911998 McAlpine
`911998 Nogales et al.
`911998 Cooper et al.
`911998 Van Seters et al.
`911998 Purcell et al.
`1011998 Davies et al.
`1011998 Isfeld et al.
`1211998 Gentry et al.
`211999 Oprea
`211999 Wakeland et al.
`311999 Bilansky et al.
`311999 Wade et al.
`411999 Klaus
`411999 Melzer et al.
`611999 Wang et al.
`611999 Kouloheris et al.
`611999 Thompson
`711999 Hendel et al.
`
`5,926,642 A
`5,930,830 A
`5,931,918 A
`5,935,205 A
`5,935,249 A
`5,937,169 A
`5,941,969 A
`5,941,972 A
`5,950,203 A
`5,963,876 A
`5,970,804 A
`5,978,844 A
`5,987,022 A
`5,991,299 A
`5,996,013 A
`5,996,024 A
`6,005,849 A
`6,009,478 A
`6,014,380 A
`6,014,557 A
`6,016,513 A
`6,021,446 A
`6,021,507 A
`6,026,452 A
`6,034,962 A
`6,034,963 A
`6,038,562 A
`6,041,058 A
`6,041,381 A
`6,044,438 A
`6,047,323 A
`6,047,356 A
`6,049,528 A
`6,057,863 A
`6,061,368 A
`6,065,096 A
`6,067,569 A
`6,070,200 A
`6,078,564 A
`6,078,733 A
`6,097,734 A
`6,101,555 A
`6,111,673 A
`6,115,615 A
`6,122,670 A
`6,141,701 A
`6,141,705 A
`6,145,017 A
`6,157,944 A
`6,157,955 A
`6,172,980 Bl
`6,173,333 Bl
`6,181,705 Bl
`6,202,105 Bl
`6,219,693 Bl
`6,223,242 Bl
`6,226,680 Bl
`6,233,242 Bl
`6,243,667 Bl
`6,246,683 Bl
`6,247,060 Bl
`6,279,051 Bl
`6,289,023 Bl
`6,298,403 Bl
`6,324,649 Bl
`6,334,153 B2
`6,343,345 Bl
`6,343,360 Bl
`6,345,301 Bl
`6,345,302 Bl
`6,356,951 Bl
`6,370,599 Bl
`6,385,647 Bl
`6,389,468 Bl
`6,389,479 Bl
`6,393,487 B2
`6,418,169 Bl
`6,421,742 Bl
`6,421,753 Bl
`
`7/1999 Favor
`7/1999 Mendelson et al.
`8/1999 Row et al.
`8/1999 Murayama et al.
`8/1999 Stern et al.
`8/1999 Connery et al.
`8/1999 Ram et al.
`8/1999 Hoese et al.
`9/1999 Stakuis et al.
`10/1999 Manssen et al.
`10/1999 Robbat, Jr.
`1111999 Tsuchiya et al.
`1111999 Geiger et al.
`1111999 Radogna et al.
`1111999 Delp et al.
`1111999 Blumenau
`12/1999 Roach et al.
`12/1999 Panner et al.
`112000 Hendel et al.
`112000 Morton et al.
`112000 Lowe
`212000 Gentry et al.
`212000 Chen
`212000 Pitts
`3/2000 Ohno et al.
`3/2000 Minami etal.
`3/2000 Anjur et al.
`3/2000 Flanders et al.
`3/2000 Hoese
`3/2000 Olnowich
`4/2000 Krause
`4/2000 Anderson et al.
`4/2000 Hendel et al.
`5/2000 Olarig
`5/2000 Hitzelberger
`5/2000 Dayet al.
`5/2000 Khakietal.
`5/2000 Gates et al.
`6/2000 Lakshman et al.
`6/2000 Osborne
`8/2000 Gotesman et al.
`8/2000 Goshey et al.
`8/2000 Chang et al.
`912000 Ota et al.
`912000 Bennett et al.
`1012000 Whitney
`1012000 Anand et al.
`1112000 Ghaffari
`1212000 Pedersen
`1212000 Narad et al.
`112001 Flanders et al.
`112001 Jolitz et al.
`112001 Branstad et al.
`3/2001 Gates et al.
`4/2001 Napolitano et al.
`4/2001 Sheafor et al.
`5/2001 Boucher et al.
`5/2001 Mayer et al.
`6/2001 Kerr et al.
`6/2001 Connery et al.
`6/2001 Boucher et al.
`8/2001 Gates et al.
`9/2001 Dowling et al.
`10/2001 Suri et al.
`1112001 Eyres et al.
`12/2001 Boucher et al.
`112002 Hilla et al.
`112002 Feinleib
`212002 Burns et al.
`212002 Bennett et al.
`3/2002 Gentry et al.
`4/2002 Anand et al.
`5/2002 Willis et al.
`5/2002 Muller et al.
`5/2002 Boucher et al.
`5/2002 Boucher et al.
`7/2002 Datari
`7/2002 Tillier
`7/2002 Hoese et al.
`
`

`

`6,427,169 Bl
`6,427,171 Bl
`6,427,173 Bl
`6,434,620 Bl
`6,434,651 Bl
`6,449,656 Bl
`6,452,915 Bl
`6,453,360 Bl
`6,453,406 Bl
`6,470,415 Bl
`6,473,425 Bl
`6,480,489 Bl
`6,483,804 Bl
`6,487,202 Bl
`6,487,654 B2
`6,490,631 Bl
`6,502,144 Bl
`6,523,119 B2
`6,526,446 Bl
`6,542,504 Bl
`6,570,884 Bl
`6,591,302 B2
`6,591,310 Bl
`6,594,261 Bl
`6,631,484 Bl
`6,648,611 B2
`6,650,640 Bl
`6,657,757 Bl
`6,658,480 B2
`6,678,283 Bl
`6,681,364 Bl
`6,683,851 Bl
`6,687,758 B2
`6,697,366 Bl
`6,697,868 B2
`6,751,665 B2
`6,757,746 B2
`6,765,901 Bl
`6,807,581 Bl
`6,842,896 Bl
`6,862,264 Bl
`6,912,522 B2
`6,938,092 B2
`6,941,386 B2
`6,965,941 B2
`6,976,148 B2
`6,996,070 B2
`7,016,361 B2
`7,042,898 B2
`7,047,320 B2
`7,073,196 Bl
`7,076,568 B2
`7,089,326 B2
`7,093,099 B2
`7,124,205 B2
`7,133,940 B2
`7,167,926 Bl
`7,167,927 B2
`7,174,393 B2
`7,181,531 B2
`7,185,266 B2
`7,187,679 B2
`7,191,241 B2
`7,191,318 B2
`7,237,036 B2
`7,254,696 B2
`7,260,518 B2
`7,283,522 B2
`7,284,070 B2
`7,287,092 B2
`7,337,241 B2
`7,461,160 B2
`7,472,156 B2
`7,496,689 B2
`7,502,869 B2
`7,519,699 B2
`7,543,087 B2
`7,584,260 B2
`7,620,726 B2
`
`7/2002 Elzur
`7/2002 Craft et al.
`7/2002 Boucher et al.
`8/2002 Boucher et al.
`8/2002 Gentry, Jr.
`912002 Elzur et al.
`912002 Jorgensen
`912002 Muller et al.
`912002 Sarnikowski et al.
`1012002 Starr et al.
`10/2002 Bellaton et al.
`1112002 Muller et al.
`1112002 Muller et al.
`1112002 Klausmeier et al.
`1112002 Dowling
`1212002 Teich et al.
`1212002 Accarie
`212003 Pav lin et al.
`212003 Yang etal.
`4/2003 Mahler et al.
`5/2003 Connery et al.
`7/2003 Boucher et al.
`7/2003 Johnson
`7/2003 Boura et al.
`1012003 Born
`1112003 Morse et al.
`1112003 Muller et al.
`1212003 Chang et al.
`1212003 Boucher et al.
`112004 Teplitsky
`112004 Calvignac et al.
`112004 Willkie etal.
`212004 Craft et al.
`212004 Kim
`212004 Craft et al.
`6/2004 Phil bri ck et al.
`6/2004 Boucher et al.
`7/2004 Johnson et al.
`1012004 Starr et al.
`112005 Redding et al.
`3/2005 Moura et al.
`6/2005 Edgar
`8/2005 Burns
`912005 Craft et al.
`1112005 Boucher et al.
`1212005 Arimilli et al.
`212006 Starr et al.
`3/2006 Swonket al.
`5/2006 Blightman et al.
`5/2006 Arimilli et al.
`7/2006 Dowdetal.
`7/2006 Phil bri ck et al.
`8/2006 Boucher et al.
`8/2006 Bodas et al.
`1012006 Craft et al.
`1112006 Blightman et al.
`112007 Boucher et al.
`112007 Phil bri ck et al.
`212007 Boucher et al.
`212007 Pinkerton et al.
`212007 Blightman et al.
`3/2007 Dally et al.
`3/2007 Boucher et al.
`3/2007 Tripathyet al.
`6/2007 Boucher et al.
`8/2007 Mittal et al.
`8/2007 Kerr et al.
`10/2007 Siddabathuni
`10/2007 Boucher et al.
`10/2007 Sharp
`212008 Boucher et al.
`1212008 Boucher et al.
`1212008 Phil bri ck et al.
`212009 Sharp et al.
`3/2009 Boucher et al.
`4/2009 Jain
`6/2009 Phil bri ck et al.
`912009 Craft et al.
`1112009 Craft et al.
`
`US 8,805,948 B2
`Page 3
`
`7,627,001 B2
`7,627,684 B2
`7,640,364 B2
`7,664,868 B2
`7,664,883 B2
`7,673,072 B2
`7,694,024 B2
`7,738,200 B2
`200110004354 Al
`200110013059 Al
`200110014892 Al
`200110014954 Al
`200110025315 Al
`200110037406 Al
`200110048681 Al
`200110053148 Al
`200210073223 Al
`200210112175 Al
`200210156927 Al
`2003/0014544 Al
`2003/0046330 Al
`2003/0066011 Al
`2003/0067903 Al
`2003/0110344 Al
`2003/0165160 Al
`2004/0010712 Al
`2004/0042458 Al
`2004/0042464 Al
`2004/0049580 Al
`2004/0049601 Al
`2004/0054814 Al
`2004/0059926 Al
`2004/0088262 Al
`2004/0153578 Al
`2004/0210795 Al
`2004/0213290 Al
`2004/0246974 Al
`2004/0249957 Al
`2005/0060538 Al
`2005/0144300 Al
`2006/0133386 Al
`2006/0248208 Al
`2007/0083682 Al
`2007/0140240 Al
`2008/0043732 Al
`2008/0170501 Al
`2008/0209084 Al
`2008/0240 III Al
`200910063696 Al
`
`1212009 Craft et al.
`1212009 Boucher et al.
`1212009 Craft et al.
`212010 Boucher et al.
`212010 Craft et al.
`3/2010 Boucher et al.
`4/2010 Phil bri ck et al.
`6/2010 Jones et al.
`6/2001 Jolitz
`8/2001 Dawson etal.
`8/2001 Gaither et al.
`8/2001 Purcell et al.
`9/2001 Jolitz
`1112001 Phil bri ck et al.
`12/2001 Bilic et al.
`12/2001 Bilic et al.
`6/2002 Darnell et al.
`8/2002 Makofka et al.
`1012002 Boucher et al.
`112003 Pettey
`3/2003 Hayes
`4/2003 Oren
`4/2003 Jorgensen
`6/2003 Szezepanek et al.
`912003 Minami etal.
`112004 Hui et al.
`3/2004 Elzu
`3/2004 Elzur et al.
`3/2004 Boyd et al.
`3/2004 Boyd et al.
`3/2004 McDaniel
`3/2004 Angelo et al.
`5/2004 Boucher et al.
`8/2004 Elzur
`1012004 Anderson
`1012004 Johnson et al.
`1212004 Gyugyi etal.
`1212004 Ekis et al.
`3/2005 Beverly
`6/2005 Craft et al.
`6/2006 McCormack et al.
`1112006 Walbeck et al.
`4/2007 Bartley et al.
`6/2007 Dally et al.
`212008 Desai et al.
`712008 Patel et al.
`8/2008 Wang etal.
`1012008 Gadelrab
`3/2009 Wang etal.
`
`FOREIGN PATENT DOCUMENTS
`
`574140 Al
`EP
`WO 98119412
`WO
`WO 98/50852
`WO
`WO 99104343
`WO
`WO 99/65219
`WO
`WO
`PCTlUS98/24943
`WO 00/13091
`WO
`WO 01104770
`WO
`WO 01105107
`WO
`WO 01105116
`WO
`WO 01105123
`WO
`WO 01140960
`WO
`WO WO 2007/130476
`
`12/1993
`5/1998
`1111998
`111999
`12/1999
`3/2000
`312000
`112001
`112001
`112001
`112001
`6/2001
`1112007
`
`OTHER PUBLICATIONS
`
`U.S. Appl. No. 601053,240, filed Jul. 18, 1997, Jolitz et al.
`Form 10-K for Exelan, Inc., for the fiscal year ending Dec. 31, 1987
`(10 pages).
`Form 10-K for Exelan, Inc., for the fiscal year ending Dec. 31, 1988
`(10 pages).
`Internet pages entitled: Technical White Paper-Xpoint 's Disk to
`LAN Acceleration Solution for Windows NT Server, printed Jun. 5,
`1997.
`Jato Technologies Internet pages entitled: Network Accelerator Chip
`Architecture, twelve-slide presentation, printed Aug. 19, 1998.
`
`

`

`US 8,805,948 B2
`Page 4
`
`EETIMES article entitled: Enterprise System Uses Flexible Spec,
`dated Aug. 10, 1998, printed Nov. 25, 1998.
`Internet pages entitled iReady About Us and iReady Products, printed
`Nov. 25, 1998.
`Internet pages entitled: Smart Ethernet Network Interface Cards
`which Berend Ozceri is developing, and Internet pages entitled:
`Hardware Assisted Protocol Processing, which Eugene Feinberg is
`working on, printed Nov. 25, 1998.
`Internet pages of Xaqti corporation entitled: GigaPower Protocol
`Processor Product review, printed Nov. 25, 2009.
`Internet pages entitled: DART: Fast Application Level Networking
`via Data-copy Avoidance, by Robert J. Walsh, printed Jun. 3, 1999.
`Internet pagees of InterProphet entitled: Frequently Asked Ques(cid:173)
`tions, by Lynne Jolitz, printed Jun. 14,2000.
`Andrew S. Tanenbaum, Computer Networks, Third Edition, 1996,
`ISBN: 0-13-349945-6.
`iReady Corporation, article entitled "The 1-1000 Internet Tuner", 2
`pages, date unknown.
`iReady News Archive article entitled "Seiko Instruments Inc. {SII)
`Introduces World's 2 First Internet-Ready Intelligent LCD Modules
`Based on iReady Technology," Santa Clara, CA and Chiba, Japan,
`Oct. 26, 1998. 2 pages, printed Nov. 2, 1998.
`WindRiver article entitled "Tornado: For Intelligent Network Accel(cid:173)
`eration", copyright Wind River Systems, 2001, 2 pages.
`WindRiver White Paper entitled "Complete TCP/IP Offload for
`High-Speed Ethernet Networks", Copyright Wind River Systems,
`2002, 7 pages.
`Intel article entitled "Solving Server Bottlenecks with Intel Server
`Adapters", Copyright Intel Corporation, 1999, 8 pages.
`Chandranmenon, Girish P. et a!. "Trading Packet Headers for Packet
`Processing." IEEEI ACM Transactions on Networking. vo!' 4, No.2.
`Apr. 1996. pp. 141-152.
`Starr, David D. et al. "Intelligent Network Storage Interface Device."
`U.S. App!. No. 09/675,700, filed Sep. 29, 2000.
`Boucher, Laurence B. et a!. "Intelligent Network Interface System
`and Method for Accelerated Protocol Processing." U.S. App!. No.
`09/692,561, filed Oct. 18,2000.
`Starr, Daryl D., et a!. "Intelligient Network Storage Interface Sys(cid:173)
`tem." U.S. App!. No. 101261,051, filed Sep. 30, 2002.
`Merritt, Rick. "Ethernet Interconnect Outpacing Infiniband at Inte!."
`EE Times article. Sep. 11, 2002. 9 pages.
`Craft, Peter K. et a!. "TCP Offload Device that Batches Session Layer
`Headers to Reduce Interrupts as Well as CPU Copies."U.S. App!. No.
`12/581,342, filed Oct. 19,2009.
`Craft, Peter K. et al. "TCP Offload Send Optimization." U.S. App!.
`No. 12/504,021, filed Ju!. 16,2009.
`Philbrick, Clive M. et a!. "Freeing Transmit Memory on a Network
`Interface Device Prior to Receiving an Acknowledgment That Trans(cid:173)
`mit Data Has Been Received by a Remote Device." U.S. App!. No.
`12/470,980, filed May 22, 2009.
`Starr, Daryl D. et a!. "Accelerating Data Transfer in a Virtual Com(cid:173)
`puter System with Tightly Coupled TCP Connections." U.S. App!.
`No. 12/410,366, filed Mar. 24, 2009.
`Boucher, Laurence B. et al. "Obtaining a Destination Address So
`That a Network Interface Device Can Write Network Data Without
`Headers Directly Into Host Memory." U.S. App!. No. 12/325,941,
`filed Dec. 1, 2008.
`Boucher, Laurence B. et a!. "Enabling an Enhanced Function of an
`Electronic Device." U.S. App!. No. 111985,948, filed Nov. 19,2007.
`Starr, Daryl D. et a!. "Network Interface Device With 10 Gb/s Full(cid:173)
`Duplex Transfer Rate." U.S. App!. No. 111799,720, filed May 1,
`2007.
`Craft, Peter K. et al. "Peripheral Device That DMAS the Same Data
`to Different Locations in a Computer." U.S. App!. No. 111788,719,
`filed Apr. 19,2007.
`Boucher, Laurence B. et a!. "TCP/IP Offload Network Interface
`Device." U.S. App!. No. 111701,705, filed Feb. 2, 2007.
`Starr, Daryl D. et a!. "TCP/IP Offload Device With Reduced Sequen(cid:173)
`tial Processing." U.S. App!. No. 111348,810, filed Feb. 6, 2006.
`Craft, Peter K. et al. "Transferring Control of TCP Connections
`Between Hierarchy of Processing Mechanisms." U.S. App!. No.
`111249,006, filed Oct. 11,2005.
`
`Boucher, Laurence B. et al. "Network Interface Device That Can
`Transfer Control of a TCP Connection to a Host CPU." U.S. App!.
`No. 111029,863, filed Jan. 4, 2005.
`Craft, Peter K. et a!. "Protocol Stack That Offloads a TCP Connection
`From a Host Computer to a Network Interface Device." U.S. App!.
`No, filed Dec. 30, 2004.
`Craft, Peter K. et a!. "Protocol Stack That Offloads a TCP Connection
`From a Host Computer to a Network Interface Device." U.S. App!.
`No. 111016,642, filed Dec. 16,2004.
`Boucher, Laurence B. et a!. "Method and Apparatus for Dynamic
`Packet Batching With a High Performance Network Interface." U.S.
`App!. No. 0/678,336, filed Oct. 3, 2003.
`Philbrick Clive M. et a!. "Method and Apparatus for Data Re-Assem(cid:173)
`bly with a High Performance Network Interface." U.S. App!. No.
`10/634,062, filed Aug. 4, 2003.
`Boucher, Laurence B. et a!. "High Network Interface Device System
`for Accelerated Communication." U.S. App!. No. 10/601,237, filed
`Jun. 19, 2003.
`Boucher, Laurence B. et a!. "Method and Apparatus for Distributing
`Network Traffic Processing on a Multiprocessor Computer." U.S.
`App!. No. 10/438,719, filed May 14, 2003.
`Boucher, LaurenceB. eta!. "Parsing a Packet Header." U.S. App!. No.
`101277,604, filed Oct. 18,2002.
`Internet pages entitled "Hardware Assisted Protocol Processing",
`(which Eugene Feinber is working on), 1 page, printed Nov. 25,1998.
`Zilog product Brief entitled "Z85C30 CMOS SCC Serial Commu(cid:173)
`nication Controller", Zilog Inc., 3 pages, 1997.
`Internet pages of Xpoint Technologies, Inc. entitled "Smart LAN
`Work Requests", 5 pages, printed Dec. 19, 1997.
`Internet pages entitled: Asante and 100BASE-T Fast Ethernet. 7
`Pages, printed May 27, 1997.
`the Paragon XP/S-A7
`to
`Internet pages entitled: A Guide
`Supercomputer at Indiana University. 13 pages printed Dec. 21,
`1998.
`Richard Stevens, "TCP/IP Illustrated, vo!' 1, The Protocols", pp.
`325-326 (1994).
`Internet pages entitled: Northridge/Southbridge vs. Intel Hub Archi(cid:173)
`tecture, 4 pages, printed Feb. 19, 2001.
`Gigabit Ethernet Technical Brief, Achieving End-to-End Perfor(cid:173)
`mance. Alteon Networks, Inc., First Edition, Sep. 1996.
`Internet pages directed to Technical Brief on Alteon Ethernet Gigabit
`NIC technology, www.alteon.com.14pages.printed Mar. 15, 1997.
`VIA Technologies, Inc. article entitled "VT8501 Apollo MVP4", pp.
`i-iv, 1-11, cover and copyright page, revision 1.3, Feb. 1,2000.
`iReady News Archives article entitled "iReady Rounding Out Man(cid:173)
`agement Team with Two Key Executives", http://www.ireadyco.
`corniarchives/keyexec.htrnl, 2 pages, printed Nov. 28, 1998.
`"Toshiba Delivers First Chips to Make Consumer Devices Internet(cid:173)
`Ready Based on iReady's Design," Press Release Oct. 1998,3 pages,
`printed Nov. 28, 1998.
`Internet pages from iReady Products, web site: http://www.ireadyco.
`corniproducts,htrnl, 2 pages, downloaded Nov. 25, 1998.
`iReady News Archives, Toshiba, iReady shipping Internet chip, I
`page, printed Nov. 25, 1998.
`Interprophet article entitled "Technology", http://www.interprophet.
`cornitechnology.htrnl, 17 ages, printed Mar. 1,2000.
`iReady Corporation, article entitled "The 1-1000 Internet Tuner", 2
`pages, date unknown.
`iReady article entitled "About Us Introduction", Internet pages from
`http://wwwiReadyco.comiabout.htrnl.3pages.printed Nov. 25,
`1998.
`iReady News Archive article entitled "Revolutionary Approach to
`Consumer Electronics Internet Connectivity Funded", San Jose, CA,
`Nov. 20, 1997.2 pages, printed Nov. 2, 1998.
`iReady News Archive article entitled "Seiko Instruments Inc. (SII)
`Introduces World's First Internet-Ready Intelligent LCD Modules
`Based on iReady Technology," Santa Clara, CA and Chiba, Japan,
`Oct. 26, 1998.2 pages, printed Nov. 2, 1998.
`NEWSwatch article entitled "iReady internet Tuner to Web Enable
`Devices", Tuesday, Nov. 5, 1996, printed Nov. 2, 1998,2 pages.
`EETimes article entitled "Tuner for Toshiba, Toshiba Taps iReady for
`Internet Tuner", by David Lammers, 2 pages, printed Nov. 2, 1998.
`
`

`

`US 8,805,948 B2
`Page 5
`
`"Comparison of Novell Netware and TCP/IP Protocol Architec(cid:173)
`tures", by J.S. Carbone, 19 pages, printed Apr. 10, 1998.
`Adaptec article entitled "AEA-71IOC-A DuraSAN product", 11
`pages, printed Oct. 1, 2001.
`iSCSI HBA article entitled "iSCSI and 2Gigabit fibre Channel Host
`Bus Adapters from Emulex, QLogic, Adaptec, JNI", 8 pages, printed
`Oct. 1,2001.
`iSCSI HBA article entitled "FCE-3210/6410 32 and 64-bit PCI-to(cid:173)
`Fibre Channel HBA", 6 pages, printed Oct. 1, 2001.
`iSCSI HBA article entitled "iSCSI Sorage", 6 pages, printed Oct. 1,
`2001.
`"Two-Way TCP Traffic Over Rate Controlled Channels: Effects and
`Analysis", by Kalampoukas et ai., IEEE Transactions on Network(cid:173)
`ing, voi. 6, No.6, Dec. 1998.
`IReady News article entitled "Toshiba Delivers First Chips to Make
`Consumer Devices Internet-Ready Based on iReady Design", Santa
`Clara, CA and Tokyo, Japan, Oct. 14, 1998, printed Nov. 2, 1998.
`U.S. Appi. No. 08/964,304, by Napolitano, et ai., entitled "File Array
`Storage Architecture", filed Nov. 4, 1997.
`"File System Design for an NFS File Server Appliance", Article by D.
`Hitz, et ai., 13 pages.
`Adaptec Press Release article entitled "Adaptec Announces
`EtherStorage Technology", 2 pages, May 4, 2000, pnnted Jun. 14,
`2000.
`Adaptec article entitled "EtherStorage Frequently Asked Questions",
`5 pages, printed Jui. 19,2000.
`Adaptec article entitled "EtherStorage White Paper", 7 pages, printed
`Jui. 19,2000.
`CIBC World Markets article entitled "Computers; Storage", by J.
`Bertino et al., 9 pages, dated Aug. 7, 2000.
`Merrill Lynch article entitled "Storage Futures", by S. Milunovich,
`22 pages, dated May 10, 2000.
`CBS Market Watch article entitled "Montreal Start-Up Battles Data
`Storage Botttieneck", by S. Taylor, dated Mar. 5, 2000, 2 pages,
`printed Mar. 7, 2000.
`Internet-draft article entitled "SCSI/TCP (SCSI over TCP)", by J.
`Satran et al., 38 pages, dated Feb. 2000, printed May 19, 2000.
`Article entitled LRP. A New Network Subsystem Architecture for
`Server Systems:, by P. Druschel and G. Banga, Rice University, 15
`pages.
`Internet RFC/STD/FYI/BCP Archives article with heading
`"RFC2140" entitled "TCP control Block Interdependence", web
`address http://www.faqs.org/rfcs/rfc2140.htrnl.9pages.printed Sep.
`20,2002.
`I. Heizer et ai., Common Internet File System Protocol (CIFS/1.0),
`Internet Draft, Internet Engineering Task Force (IETF), Jun. 13,
`1996.
`David D. Clark et ai., NETBLT: A Bulk Data Transfer Protocol,
`Request for Comments (RFC) 998, Mar. 1987.
`K. Sollins, The TFTP Protocol (Revision 2), Request for Comments
`(RFC) 1350, Jui. 1992.
`W. David Schwaderer, C Progranuner's Guide to NetBIOS, IPX, and
`SPX, 1992, pp. 322-323.
`Internet pages entitled Technical White Paper-Xpoint's Disk to LAN
`Acceleration Solution for Windows NT Server, printed Jun. 5, 1997,
`15 pages.
`.
`.
`Jato Technologies article entitled Network Accelerator Chip Archi(cid:173)
`tecture, twelve-slide presentation, printed Aug. 19, 1998, 13 pages.
`EETimes article entitled Enterprise System Uses Flexible Spec,
`dated Aug. 1 0, 1998, printed Nov. 25, 1998, 3 pages.
`Internet pages entitled "Smart Ethernet Network Interface Cards",
`which Berend Ozceri is developing, printed Nov. 25, 1998,2 pages.
`Internet pages of Xaqti corporation entitled "GigaPower Protocol
`Processor Product Review," printed Nov. 25, 1999,4 pages.
`U.S. Appi. No. 601283,896, Titled: CRC Calculations for Out of
`Order PUDs, filed Apr. 12, 2003, Inventor: Amit Oren, Assignee:
`Siliquent Technologies Ltd.
`Internet pages entitled "DART: Fast Application Level Networking
`via Data-Copy Avoidance," by Robert J. Walsh, printed Jun. 3, 1999,
`25 pages.
`Andrew S. Tanenbaum, Computer Networks, Third Edition, 1996,
`ISBN 0-13-349945-6.
`Article from Rice University entitled "LRP: A New Network Sub(cid:173)
`system Architecture for Server Systems", by Peter Druschel and
`Gaurav Banga, 14 pages.
`
`Internet RFD/STD/FYI/BCP Archives article with heading
`"RFC2140" entitled "TCP Control Block Interdependence", web
`address http://www.faqs.org/rfcs/rfc2140.html.9pages.printed Sep.
`20,2002.
`Wind River article entitled "Tornado: For Intelligent Network Accel(cid:173)
`eration", copyright Wind River Systems, 2001, 2 pages.
`Schwaderer et al., IEEE Computer Society Press publication entitled,
`"XTP in VLSI Protocol Decomposition for ASIC Implementation",
`from 15th Conference on Local Computer Networks, 5 pages, Sep.
`30-0ct. 3, 1990.
`Beach, Bob, IEEE Computer Society Press publication entitled,
`"UitraN et: An Architecture for Gigabit Networking", from 15th Con(cid:173)
`ference on Local Computer Networks, 18 pages, Sep. 30-0ct. 3,
`1990.
`Chesson et ai., IEEE Syposium Record entitled, "The Protocol
`Engine Chipset", from Hot Chips III, 16 pages, Aug. 26-27,1991.
`Maclean et ai., IEEE Global Telecommunications Conference,
`Globecom '91, presentation entitled, "An Outboard Processor for
`High Performance Implementation of Transport Layer Protocols", 7
`pages, Dec. 2-5, 1991.
`.
`.
`Ross etai., IEEE article entitled "FXI000: Ahlgh performance smgle
`chip Gigabit Ethernet NIC", from Compean '97 Proceedings, 7
`pages, Feb. 23-26, 1997.
`Strayer et ai., "Ch. 9: The Protocol Engine" from XTP: The Transfer
`Protocol, 12 pages, Jui. 1992.
`Publication entitled "Protocol Engine Handbook", 44 pages, Oct.
`1990.
`Koufopavlou et ai., IEEE Global Telecommunications Conference,
`Globecom '92, presentation entitled, "Parallel TCP for High Perfor(cid:173)
`mance Communication Subsystems", 7 pages, Dec. 6-9, 1992.
`Lilienkamp et ai., Publication entitled "Proposed Host-Front End
`Protocol", 56 pages, Dec. 1984.
`Thia et ai. Publication entitled "High-Speed OSI Protocol Bypass
`Algorithm with Window Flow Control," Protocols for High Speed
`Networks, pp. 53-68, 1993.
`U.S. Appi. No. 601053,240, Titled: TCP/IP Network Accelerator and
`Method of Use, filed Jui. 17, 1997, Inventor: William Jolizt et ai.
`Thia et ai. Publication entitled "A Reduced Operational Protocol
`Engine (ROPE) for a multiple-layer bypass architecture," Protocols
`for High Speed Networks, pp. 224-239, 1995.
`Thia, YH. Publication entitled "High-Speed OSI Protocol Bypass
`Algorithm with Window Flow Control", Protocols for High Sgeed
`Networks, pp. 53-68, 1993.
`Thia, YH. Publication entitled"A Reduced Operational Protocol
`Engine (ROPE) for a multiple layer bypass architecture", Protocols
`for High Speed Networks, pp. 224-239, 1995.
`
`Primary Examiner - Moustafa M Meky
`(74) Attorney, Agent, or Firm - Mark Lauer; Silicon Edge
`Law Group LLP
`
`ABSTRACT
`(57)
`A system for protocol processing in a computer network .has
`an intelligent network interface card (INIC) or commumca(cid:173)
`tion processing device (CPD) associated with a host com(cid:173)
`puter. The INIC or CPD provides a fast-path that avoids host
`protocol processing for most large multipacket messages,
`greatly accelerating data communication. The INIC or CPD
`also assists the host for those message packets that are chosen
`for processing by host software layers. A communication
`control block (CCB) for a message is defined that allows
`DMA controllers of the INIC to move data, free of headers,
`directly to or from a destination or source in the host. ~he
`CCB can be passed back to the host for message processmg
`by the host. The INIC or CPD contains hardware circuits
`configured for protocol processing that can perform that spe(cid:173)
`cific task faster than the host cpu. One embodiment includes
`a processor providing transmit, receive and management pro(cid:173)
`cessing, with full duplex communication for four fast Ether(cid:173)
`net nodes.
`
`22 Claims, 14 Drawing Sheets
`
`

`

`u.s. Patent
`
`Aug. 12,2014
`
`Sheet 1 of 14
`
`US 8,805,948 B2
`
`20
`I--------------------------l
`HOST
`I
`1_ '
`I
`- -
`
`STORAGE
`
`28
`
`35
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`30 I
`I
`
`22
`
`REMOTE
`I-lOST
`
`25
`
`CPD
`
`I
`I
`I
`I
`'------
`
`FIG. 1
`
`46
`
`(
`
`CPU
`
`33
`
`~ STORAGE
`
`(50
`
`42,,-
`
`UPPER
`LAYER
`UPPER LAYER 48,
`INTERFACE
`
`CONTEXT "J
`ro TRANSPORT
`
`54
`
`52
`"--
`
`38"""'-
`
`36
`
`NETWORK
`
`DATA LINK
`
`~
`35
`
`~8
`
`\
`
`44
`
`56
`~
`
`30~
`
`INIC/CPD
`
`FIG. 2
`
`

`

`u.s. Patent
`
`Aug. 12,2014
`
`Sheet 2 of 14
`
`US 8,805,948 B2
`
`RECEIVE PACKET
`FROM NETWORK
`BYCPD
`
`VALIDATE PACKET,
`SUMMARIZE
`HEADERS
`
`47
`
`57
`
`61
`
`5~
`
`FAST PATH
`CANDIDATE?
`
`NO
`
`SEND PACKET TO
`STACK FOR SLOW-
`PATH PROCESSING
`
`67
`
`YES
`
`53
`
`65
`
`MATCH WITH
`CCB?
`
`NO
`
`SEND PACKET TO
`STACK FOR SLOW-
`PATH PROCESSING
`
`69
`
`SEND TO
`DESTINATION
`IN HOST VIA
`FAST-PATH
`
`FIG. 3
`
`CREATE CCB FOR
`:MESSAGE
`
`51
`
`

`

`u.s. Patent
`
`Aug. 12,2014
`
`Sheet 3 of 14
`
`US 8,805,948 B2
`
`REMOTE
`HOST
`
`22
`
`25
`
`32
`
`FIG.4A
`
`42
`
`1
`40 35 1
`38 ~
`36
`
`SESS
`TRANS
`NETW
`DLTNK
`~44
`
`SESS
`~-L-f~~~---~I In
`66 :
`'---.l~=~=~+__I TRANS
`:
`40 35
`:
`:
`'----1
`NETW
`38
`36
`DUNK
`L ______ ~
`
` -------
`
`22
`
`REMOTE
`25
`HOST
`L - -__ ~-----'---------',,.'RECETVE LOGIC
`32J-------
`FIG.4B
`
`56
`
`42
`40
`38
`36
`
`~44
`
`SESS
`TRANS
`NETW
`DUNK
`~44
`
`1-------1
`:
`1 2
`:
`35
`1
`1
`~ 66 70:
`
`1
`1
`L... ______ J
`
`82
`66
`
`1 L __ __ __ ..J
`
`22
`
`REMOTE
`HOST
`
`25
`
`22
`
`REMOTE
`HOST
`
`32
`
`FIG.4C
`
`FIG.4D
`
`

`

`u.s. Patent
`
`Aug. 12,2014
`
`Sheet 4 of 14
`
`US 8,805,948 B2
`
`REMOTE
`HOST
`
`22
`
`25
`
`99
`
`--1
`
`42
`92
`1~~L-i-~SE~S~S~r---L-~~l :
`TRANS
`~---' 1
`1
`NETW
`90 1
`1
`1
`1
`DLnITZ
`---r-- J
`35----
`
`/
`
`~44
`
`FIG. 5
`
`15l
`r- ~-- --~l--------\-- -,
`
`150",
`
`170
`
`185
`
`1
`1
`1
`1
`1
`1
`1
`
`I.
`
`1
`
`1
`1
`1
`1
`I
`I HARDWARE LOGIC
`I
`- - - ~- -
`L ____
`171
`
`PROCESSOR
`
`S
`
`-
`
`-
`
`- - - - - - -
`
`t'----- 155
`'If
`
`FIG. 6
`
`160~
`SLOW-PATH
`C 157
`
`152-,{----------------,
`I
`I
`FAST-PATH
`I
`1 :~ SOURCEIDEST
`I
`168 -t
`I
`I
`I
`I
`166 ~..L
`1
`I
`I
`I
`I
`1
`1
`1
`I
`1
`1
`1
`1
`1
`1
`
`1
`
`1
`
`1
`1
`
`164--t
`
`1
`
`162--i--
`
`1
`
`1
`
`,'1.
`
`APPLICA nON
`
`TRANSPORT
`
`NETWORK
`
`DATA LINK
`
`.! -----~1-58-----
`
`

`

`u.s. Patent
`
`Aug. 12,2014
`
`Sheet 5 of 14
`
`US 8,805,948 B2
`
`MEDIA ACCESS
`~172
`CONTROLLER
`
`I
`
`ASSEMBLY ~174
`REGISTER
`
`)178
`
`76
`)1
`
`, ,
`
`FLYBY
`SEQUENCER
`
`PACKET
`CONTROL
`SEQUENCER
`
`MUL TIPLEXOR ~180
`
`,
`
`) 182
`
`) 183
`
`SRAM
`
`SRAM
`CONTROL
`
`DRAM CONTROL
`
`( 186
`
`,
`
`DRAM ~188
`
`QUEUE
`184~
`MANAGER
`
`FIG. 7
`
`

`

`u.s. Patent
`
`Aug. 12,2014
`
`Sheet 6 of 14
`
`US 8,805,948 B2
`
`PACKET
`176 ________ CONTROL
`SEQUENCER
`
`f----~
`
`ASSEMBLY
`REGISTER
`
`178
`
`r------ 191
`MAC
`1-00II.1--------;
`SEQUENCER -
`
`,
`
`NETWORK --- 192
`14 - - - - - - -1
`SEQUENCER -
`
`TRANSPORT r----- 194
`t . - - - - - - - - - i
`SEQUENCER
`
`1r
`
`SESSION