111111111111111111111111111111111111111111111111111111111111111111111111111
`US008645558B2
`
`c12) United States Patent
`Pope et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,645,558 B2
`Feb.4,2014
`
`(54) RECEPTION ACCORDING TO A DATA
`TRANSFER PROTOCOL OF DATA DIRECTED
`TO ANY OF A PLURALITY OF DESTINATION
`ENTITIES FOR DATA EXTRACTION
`
`(75)
`
`Inventors: Steven Leslie Pope, Cambridge (GB);
`Derek Edward Roberts, Cambridge
`(GB); David James Riddoch,
`Cambridge (GB); Greg Law, Cambridge
`(GB); Steve Grantham, Huntingdon
`(GB); Matthew Slattery, Cambridge
`(GB)
`
`(73) Assignee: Solarflare Communications, Inc.,
`Irvine, CA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1191 days.
`
`(21) Appl. No.:
`
`11/916,973
`
`(22) PCTFiled:
`
`Jun.15,2006
`
`(86) PCTNo.:
`
`PCT /GB2006/002202
`
`§ 371 (c)(l),
`(2), ( 4) Date: Nov.14, 2008
`
`(87) PCT Pub. No.: W02006/134373
`
`PCT Pub. Date: Dec. 21, 2006
`
`(65)
`
`Prior Publication Data
`
`US 2012/0124121 Al
`
`May 17,2012
`
`(30)
`
`Foreign Application Priority Data
`
`Jun. 15, 2005
`Jun. 17, 2005
`Jun.20,2005
`Jun.20,2005
`Jun.20,2005
`Jun.20,2005
`Jun.20,2005
`Jun.20,2005
`
`(GB) ................................... 0512219.7
`(GB) ................................... 0512425.0
`(GB) ................................... 0512532.3
`(GB) ................................... 0512533.1
`(GB) ................................... 0512534.9
`(GB) ................................... 0512535.6
`(GB) ................................... 0512536.4
`(GB) ................................... 0512537.2
`
`(51)
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`Int. Cl.
`G06F 15116
`G06F 3100
`H04J3/16
`(52) U.S. Cl.
`USPC ............................ 709/230; 719/328; 370/466
`(58) Field of Classification Search
`USPC ........................... 709/230; 711/111; 719/328;
`370/466-467
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,272,599 A
`5,325,532 A
`5,946,189 A
`5,951,645 A *
`6,021,445 A *
`6,098,112 A
`6,160,554 A
`6,304,945 B1
`6,349,035 B1
`6,438,130 B1
`6,502,203 B2
`6,530,007 B2
`6,667,918 B2
`6,718,392 B1
`6,728,743 B2
`6,735,642 B2
`6,768,996 B1
`6,901,594 B1 *
`6,904,534 B2
`6,950,961 B2
`6,978,331 B1
`6,988,262 B1 *
`7,093,158 B2
`7,099,275 B2
`7,103,626 B1
`7,103,744 B2
`7,136,397 B2
`7,143,412 B2
`7,149,227 B2
`7,151,744 B2
`7,178,062 B1
`7,216,225 B2
`7,240,350 B1
`7,245,627 B2
`7,254,237 B1
`
`12/1993 Koenen
`6/1994 Crosswy eta!.
`8/1999 Koenen et a!.
`9/1999 Goto ............................. 709/230
`212000 Chapa ........................... 719/328
`8/2000 Ishijima et al.
`12/2000 Krause
`10/2001 Koenen
`212002 Koenen
`8/2002 Kagan eta!.
`12/2002 Barron eta!.
`3/2003 Olarig et al.
`12/2003 Leader eta!.
`4/2004 Krause
`4/2004 Shachar
`5/2004 Kagan eta!.
`7/2004 Steffens et a!.
`5/2005 Cain eta!. ..................... 719/310
`6/2005 Koenen
`9/2005 Krause et a!.
`12/2005 Kagan eta!.
`112006 Mallory eta!. ............... 717/127
`8/2006 Barron et a!.
`8/2006 Sarkinen et a!.
`9/2006 Recio eta!.
`9/2006 Garcia eta!.
`1112006 Sharma
`1112006 Koenen
`12/2006 Stoler eta!.
`12/2006 Sarkinen eta!.
`2/2007 Dice
`5/2007 Haviv eta!.
`7/2007 Eberhard eta!.
`7/2007 Goldenberg eta!.
`8/2007 Jacobson et al.
`
`18a
`
`18b
`
`16a
`
`~
`
`10a
`
`11b,}
`
`10b
`
`001
`
`

`
`US 8,645,558 B2
`Page 2
`
`10/2007 Fiedler
`7,285,996 B2
`112008 Jacobson et al.
`7,316,017 B1
`3/2008 Haviv
`7,346,702 B2
`6/2008 Jacobson et al.
`7,386,619 B1
`7/2008 Modi eta!.
`7,403,535 B2
`7/2008 Krause eta!.
`7,404,190 B2
`3/2009 Barron eta!.
`7,502,826 B2
`3/2009 Hanes eta!.
`7,509,355 B2
`3/2009 Wainner et al.
`7,509,491 B1
`4/2009 Smelloyetal.
`7,518,164 B2
`6/2009 Teisberg et a!.
`7,551,614 B2
`6/2009 Modi eta!.
`7,554,993 B2
`8/2009 Fiedler
`7,573,967 B2
`8/2009 Hudson eta!.
`7,580,415 B2
`8/2009 Fiedler
`7,580,495 B2
`1112009 Chadalapaka et al.
`7,617,376 B2
`12/2009 Goldenberg eta!.
`7,631,106 B2
`112010 McMahan et a!.
`7,650,386 B2
`112010 Kagan eta!.
`7,653,754 B2
`3/2010 Santiago et a!.
`7,688,853 B2
`7/2010 Hilland et al.
`7,757,232 B2
`9/2010 Kagan eta!.
`7,801,027 B2
`9/2010 Oved
`7,802,071 B2
`10/2010 Fiedler
`7,813,460 B2
`1112010 Sharma eta!.
`7,827,442 B2
`1112010 Sarkinen et a!.
`7,835,375 B2
`12/2010 Oved
`7,848,322 B2
`12/2010 Cripe et al.
`7,856,488 B2
`112011 Oved
`7,864,787 B2
`3/2011 Krause eta!.
`7,904,576 B2
`4/2011 Haviv
`7,921,178 B2
`4/2011 Kagan eta!.
`7,929,539 B2
`4/2011 Kagan eta!.
`7,930,437 B2
`5/2011 Rephaeli et a!.
`7,934,959 B2
`7/2011 Buskirk et a!.
`7,978,606 B2
`8/2011 Hare!
`8,000,336 B2
`5/2002 Bloch eta!.
`2002/0059052 A1
`9/2002 Hutton eta!.
`2002/0129293 A1
`10/2002 Hudson
`2002/0140985 A1
`10/2002 Hanes eta!.
`2002/0156784 A1
`2002/0183864 A1 * 12/2002 Ape! eta!. ...................... 700/14
`2003/0007165 A1
`112003 Hudson
`2003/0055866 A1
`3/2003 Leclair et a!.
`2003/0058459 A1
`3/2003 Wu eta!.
`2003/0063299 A1
`4/2003 Cowanet a!.
`2003/0065856 A1
`4/2003 Kagan eta!.
`2003/0081060 A1
`5/2003 Zeng eta!.
`2003/0114949 A1
`6/2003 Armstrong et a!.
`2003/0163589 A1
`8/2003 Bunce et al.
`2003/0172330 A1
`9/2003 Barron eta!.
`2003/0191786 A1
`10/2003 Matson
`2003/0202043 A1
`10/2003 Zeng eta!.
`2003/0202536 A1
`10/2003 Foster eta!.
`2003/0214677 A1
`1112003 Bhaskar et a!.
`2004/0071250 A1
`4/2004 Bunton
`2004/0117793 A1
`6/2004 Shaylor
`2004/0141642 A1
`7/2004 Zeng eta!.
`2004/0190533 A1
`9/2004 Modi eta!.
`2004/0190538 A1
`9/2004 Bunton et al.
`2004/0190557 A1
`9/2004 Barron
`2004/0193734 A1
`9/2004 Barron eta!.
`2004/0193825 A1
`9/2004 Garcia eta!.
`2004/0210754 A1
`10/2004 Barron eta!.
`2004/0252685 A1
`12/2004 Kagan eta!.
`2005/0008223 A1
`112005 Zeng eta!.
`2005/0018221 A1
`112005 Zeng eta!.
`2005/0038918 A1
`2/2005 Hilland et al.
`2005/0038941 A1
`2/2005 Chadalapaka et al.
`2005/0039171 A1
`2/2005 Avakian eta!.
`2005/0039172 A1
`2/2005 Rees
`2005/0039187 A1
`2/2005 Bloom eta!.
`2005/0066333 A1
`3/2005 Krause eta!.
`2005/0172181 A1
`8/2005 Huliehel
`2005/0219278 A1
`10/2005 Hudson
`2005/0219314 A1
`10/2005 Donovan et a!.
`2005/0226250 A1
`10/2005 Makayama et al.
`2005/0231751 A1
`10/2005 Wu eta!.
`2006/0026443 A1
`2/2006 McMahan et a!.
`
`2006/0031525 A1 * 2/2006 Reeves eta!. ................. 709/227
`2006/0045098 A1
`3/2006 Krause
`2006/0126619 A1
`6/2006 Teisberg et a!.
`2006/0165074 A1
`7/2006 Modi et al.
`2006/0168400 A1 * 7/2006 Ronciak et a!. ............... 7111118
`2006/0193318 A1
`8/2006 Narasimhan eta!.
`2006/0228637 A1
`10/2006 Jackson et a!.
`2006/0248191 A1
`1112006 Hudson eta!.
`2007/0188351 A1
`8/2007 Brown eta!.
`2007/0220183 A1
`9/2007 Kagan eta!.
`2008/0024586 A1
`1/2008 Barron
`2008/0109526 A1
`5/2008 Subramanian et a!.
`2008/0115216 A1
`5/2008 Barron eta!.
`2008/0115217 A1
`5/2008 Barron eta!.
`2008/0126509 A1
`5/2008 Subramanian et a!.
`2008/0135774 A1
`6/2008 Hugers
`2008/0147828 A1
`6/2008 Enstone et a!.
`2008/0148400 A1
`6/2008 Barron eta!.
`2008/0177890 A1
`712008 Krause eta!.
`2008/0244060 A1
`10/2008 Cripe eta!.
`2008/0301406 A1
`12/2008 Jacobson et al.
`2008/0304519 A1
`12/2008 Koenen eta!.
`2009/0165003 A1
`6/2009 Jacobson et al.
`2009/0201926 A1
`8/2009 Kagan eta!.
`2009/0213856 A1
`8/2009 Paatela et a!.
`2009/0268612 A1
`10/2009 Felderman et a!.
`2009/0302923 A1
`12/2009 Smeloyet a!.
`2010/0088437 A1
`4/2010 Zahavi
`2010/0138840 A1
`6/2010 Kagan eta!.
`2010/0169880 A1
`712010 Haviv eta!.
`2010/0188140 A1
`712010 Smeloy
`2010/0189206 A1
`712010 Kagan
`2010/0265849 A1
`10/2010 Hare!
`2010/0274876 A1
`10/2010 Kagan eta!.
`201110004457 A1
`112011 Haviv eta!.
`201110010557 A1
`112011 Kagan eta!.
`201110029669 A1
`212011 Chuang eta!.
`201110029847 A1
`212011 Goldenberg et a!.
`201110044344 A1
`212011 Hudson eta!.
`201110058571 A1
`3/2011 Bloch eta!.
`201110083064 A1
`4/2011 Kagan eta!.
`201110096668 A1
`4/2011 Bloch eta!.
`201110113083 A1
`5/2011 Shahar
`201110116512 A1
`5/2011 Crupnicoff et a!.
`201110119673 A1
`5/2011 Bloch eta!.
`201110173352 A1
`7/2011 Sela eta!.
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`wo
`wo
`wo
`wo
`wo
`wo
`wo
`wo
`wo
`wo
`
`620521
`2272214
`W001148972
`W002/35838
`wo 03/107611
`W02008/127672
`W02009/136933
`W020 10/020907
`W020 10087826
`W020111043768
`W020111053305
`W020111053330
`
`3/1994
`112011
`7/2001
`5/2002
`12/2003
`10/2008
`1112009
`2/2010
`8/2010
`4/2011
`5/2011
`5/2011
`
`OTHER PUBLICATIONS
`
`Adiga, NR. et al. An Overview of the BlueGene/L Supercomputer.
`Proceedings of the 2002 ACM/IEEE conference on Supercomputing,
`Baltimore. pp. 1-22.
`Aggarwal, Vinay et a!. Workshop on Network-I/O Convergence:
`Experience, Lessons, Implications (Niceli). ACM Computer Com-
`munication Review, vol. 33, No.5, 2003. pp. 75-80.
`Allman, Mark. TCP Byte Counting Refinements. ACM Computer
`Communication Review, vol. 29, No.3, 1999. pp. 14-22.
`Anderson, Ed eta!. Performance of the CRAY T3E Multiprocessor.
`Proceedings of the 1997 ACM/IEEE conference on Supercomputing,
`San Jose, California. pp. 1-17.
`Angin, Oguz eta!. Report on the 5th IFIP Internation Workshop on
`Quality of Service (IWQOS 97). ACM Computer Communication
`Review, vol. 27, No.3, 1997. pp. 100-117.
`
`002
`
`

`
`US 8,645,558 B2
`Page 3
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Balaji, P. eta!. Head-to- Toe Evaluation of High-Performance Sockets
`Over Protocol Offload. Engines. Proceedings of the IEEE Interna(cid:173)
`tional Conference on Cluster Computing, 2005. pp. 1-10.
`Barham, Paul Ronald. Devices in a Multi-Service Operating System.
`PhD Thesis, University of Cambridge, 1996. 131 pages.
`Bhandarkar, Sumitha et a!. LTCP: Improving the Performance of
`TCP in Highspeed Networks. ACM Computer Communication
`Review, vol. 36, No. 1, 2006. pp. 41-50.
`Birrell, Andrew D. et a!. Grapevine: An Exercise in Distributed
`Computing. Communications of the ACM, vol. 25, Issue 4, 1982. pp.
`260-274.
`Black, Richard et al. Experiences of Building an ATM Switch for the
`Local Area. ACM Computer Communication Review, vol. 24, No.4,
`1994. pp. 158-167.
`Blanton, Ethan eta!. On Making TCP More Robust to Packet Reor(cid:173)
`dering. ACM Computer Communication Review, vol. 32, No. 1,
`2002. pp. 20-30.
`Boden, Nanette J. eta!. Myrinet: A Gigabit-per-Second Local-Area
`Network. Draft of paper published in IEEE Micro, vol. 15, No. 1,
`1995. pp. 29-36.
`Boggs, David R. eta!. Measured Capacity of an Ethernet: Myths and
`Reality. ACM Computer Communication Review, vol. 18, No. 4,
`1988. pp. 222-234.
`Borman, DavidA. Implementing TCPIIP on a Cray computer. ACM
`Computer Communication Review, vol. 19, No.2, 1989. pp. 11-15.
`Braden, R. eta!. Computing the Internet Checksum. ACM Computer
`Communication Review, vol. 19, No.2, 1989. pp. 86-94.
`Brakmo, Lawrence S. et al. TCP Vegas: New Techniques for Conges(cid:173)
`tion Detection and Avoidance. ACM Computer Communication
`Review, vol. 24, No.4, 1994. pp. 24-35.
`Brandriff, Robert M. eta!. Development of a TCPIIP for the IBM/3 70.
`ACM Computer Communication Review, vol. 15, No.4, 1985. pp.
`2-8.
`Buonadonna, Philip et a!. An Implementation and Analysis of the
`Virtual Interface Architecture. Proceedings of the 1998 ACM/IEEE
`conference on Supercomputing, Orlando, Florida. pp. 1-15.
`Calvert, Ken. Reflections on Network Architecture: an Active Net(cid:173)
`working Perspective. ACM Computer Communication Review, vol.
`36, No.2, 2006. pp. 27-30.
`Cerf, V. et al. Proposal for an International End-to-End Protocol.
`ACM Computer Communication Review, vol. 6 No. 1, 1976. pp.
`1-18.
`Cerf, V. et a!. A Protocol for Packet Network Intercommunication.
`IEEE Transactions on Communications, vol. COM-22, No. 5, 1974.
`pp. 637-648.
`Chang, Chi-Chao eta!. Low-Latency Communication on the IBM
`RISC System/6000 SP. Proceedings of the 1996 ACM/IEEE confer(cid:173)
`ence on Supercomputing, Pittsburgh. pp. 1-17.
`Cheriton, David R. Sirpent: A High-Performance Internetworking
`Approach. ACM Computer Communication Review, vol. 19, No.4,
`1989. pp. 158-169.
`Chesson, Dr. Gregory L. Declaration of Dr Gregory L Chesson in
`Alacritech v. Microsoft. United States District Court, Northern Dis(cid:173)
`trict California, San Francisco Division. Feb. 4, 2005. 289 pages.
`Chesson, Dr. Gregory L. The Evolution of XTP. Proceedings of the
`Third International Conference on High Speed Networking, 1991.
`pp. 1-10.
`Chiou, Derek eta!. StarT- Voyager: A Flexible Platform for Exploring
`Scalable SMP Issues. Proceedings of the 1998 ACM/IEEE confer(cid:173)
`ence on Supercomputing, Orlando, Florida. 20 pages.
`Chu, H. K. Jerry. Zero-Copy TCP in Solaris. Proceedings of the
`USENIX Annual Technical Conference 1996. 13 pages.
`Clark, David D. et a!. An Analysis of TCP Processing Overhead.
`IEEE Communications Magazine, vol. 27, No.6, 1989. pp. 23-29.
`Clark, David D. et al. Architectural Considerations for a New Gen(cid:173)
`eration of Protocols. ACM Computer Communication Review, vol.
`20, No.4, 1990. pp. 200-208.
`
`Clark, David. D. The Design Philosophy of the DARPA Internet
`Protocols. ACM Computer Communication Review, vol. 18, No.4,
`1988. pp. 106-114.
`Cohen, Danny et al. Use of message-based multicomputer compo(cid:173)
`nents to construct gigabit networks. ACM Computer Communication
`Review, vol. 23, No.4, 1993. p. 32-44.
`Cohen, Gregory et a!. ATOMIC: A High-Speed Local Communica(cid:173)
`tionArchitechture. Journal ofHigh Speed Networks. Jan. 3, 1994. pp.
`1-21.
`Cohen, Gregory eta!. ATOMIC: A Local Communication Network
`Created Through Repeated Application of Multicomputing Compo(cid:173)
`nents. Made available by authors. Jan. 10, 1992. 11 pages.
`Cooper, Eric C. et a!. Protocol Implementation on the Nectar Com(cid:173)
`munication Processor. ACM Computer Communication Review, vol.
`20, No.4, 1990. pp. 135-144.
`Crowcroft, John. 10 Networking Papers: Recommended Reading.
`ACM Computer Communication Review, vol. 36, No. 2, 2006. pp.
`51-52.
`Crowcroft, John et a!. ATM: A Retrospective on Systems Legacy.
`ACM Computer Communication Review, vol. 32, No. 5, 2002. pp.
`11-12.
`Crowley, Patrick et a!. Characterizing Processor Architectures for
`Programmable Network Interfaces. Proceedings of the 14th interna(cid:173)
`tional conference on Supercomputing, Santa Fe, New Mexico. May
`8, 2000. pp. 54-65.
`Currid, Andy. TCP Offload to the Rescue. ACM Queue, vol. 2, No.3,
`2004. pp. 60-65.
`Davie, Bruce S. A Host-Network Interface Architecture for ATM.
`ACM Computer Communication Review, vol. 21, No.4, 1991. pp.
`307-315.
`Delp, Gary S. eta!. An Analysis ofMemnet: An Experiment in High(cid:173)
`Speed Shared-Memory Local Networking. ACM Computer Commu(cid:173)
`nication Review, vol. 18, No.4, 1988. pp. 165-174.
`Dennis, Jack B. eta!. Programming Semantics for Multiprogrammed
`Computations. Communications oftheACM, vol. 9, No.3, 1966. pp.
`143-155.
`De Vivo, M. et al. Internet Vulnerabilities Related to TCPIIP and
`TITCP. ACM Computer Communication Review, vol. 29, No. 1,
`1999. pp. 81-85.
`Dickman, Lloyd. Presentation: Protocol Ojfloading vs Onloading in
`High Performance Networks. Hot Interconnects Panel, Aug. 23,
`2006. 7 pages.
`Druschel, Peter eta!. Experiences with a High-Speed Network Adap(cid:173)
`tor: A Software Perspective. ACM Computer Communication
`Review, vol. 24, No.4, 1994. pp. 2-13.
`Druschel, Peter et a!. Fbufs: A High-Bandwidth Cross-Domain
`Transfer Facility. ACM Operating Systems Review, vol. 27, Issue 5,
`1993. pp. 189-202.
`Edwards, A. et al. Experiences Implementing a High-Peiformance
`TCP in User-Space. ACM Computer Communication Review, vol.
`25, No.4, 1995. pp. 196-205.
`Edwards, A. et a!. User-Space Protocols Deliver High Performance
`to Applications on a Low-Cost Gb/s LAN. ACM Computer Commu(cid:173)
`nication Review, vol. 24, No.4, 1994. pp. 14-23.
`von Eicken, Thorsten et a!. U-Net: A User-Level Network Interface
`for Parallel and Distributed Computing. ACM Operating Systems
`Review, vol. 29, Issue 5, 1995. pp. 109-126.
`Evans, Joseph B. et a!. The End of History. IEEE TCGN Gigabit
`Networking Workshop, 2001. 10 pages.
`Falsafi, Babak et al. Application-Specific Protocols for User-Level
`Shared Memory. Proceedings of the 1994 conference on
`Supercomputing, Washington D.C. pp. 380-389.
`Feng, Wu-chun, eta!. Optimizing 10-Gigabit Ethernet for Networks
`of Workstations, Clusters, and Grids: A Case Study. Proceedings of
`the 2003 ACM/IEEE conference on Supercomputing, Phoenix, Ari(cid:173)
`zona. 13 pages.
`Feng, Wu-chun, et a!. The Failure of TCP in High-Performance
`Computational Grids. Proceedings of the 2000 ACM/IEEE confer(cid:173)
`ence on Supercomputing, Dallas, Texas. 11 pages.
`Finn, Gregory G. An Integration of Network Communication with
`Workstation Architecture. ACM Computer Communication Review,
`vol. 21, No.5, 1991. pp. 18-29.
`
`003
`
`

`
`US 8,645,558 B2
`Page 4
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Finn, Gregory G. eta!. Netstation Architecture Multi-Gigabit Work(cid:173)
`station Network Fabric. Proceedings of InterOp '94, Las Vegas,
`Nevada. pp. 1-9.
`Floyd, Sally. TCP and Explicit Congestion Notification. ACM Com(cid:173)
`puter Communication Review, vol. 24, No.5, 1994. pp. 8-23.
`Geoffray, Patrick. A Critique of RDMA. HPCWire article: http://
`www.hpcwire.com/features/17886984.html. Aug. 18, 2006. 7 pages.
`Geoffray, Patrick. From Presentation: Protocol off-loading vs on(cid:173)
`loading in high-performance networks. Hot Interconnects 2006 at
`http://www.myri.com. 5 pages.
`Hayter, Mark David. A Workstation Architecture to Support Multi(cid:173)
`media. PhD Thesis, University of Cambridge, 1993. 111 pages.
`Hayter, Mark David et al. The Desk Area Network. ACM Operating
`Systems Review, vol. 25, Issue 4, 1991. pp. 14-21.
`Hill, J. Carver. Synchronizing Processors with Memory-Content(cid:173)
`Generated Interrupts. Communications of the ACM, vol. 16, No.6,
`1973. pp. 350-351.
`Hsieh, Jenwei et a!. Architectural and Performance Evaluation of
`GigaNet and Myrinet Interconnects on Clusters of Small-Scale SMP
`Servers. Proceedings of the 2000 ACM/IEEE conference on
`Supercomputing, Dallas, Texas. pp. 1-9.
`Husbands, Parry eta!. MPI-StarT Delivering Network Performance
`to Numerical Applications. Proceedings of the 1998 ACM/IEEE
`conference on Supercomputing, Orlando, Florida. pp. 1-15.
`Jacobson, Van. Congestion Avoidance and Control. ACM Computer
`Communication Review, vol. 18, No.4, 1988. pp. 314-329.
`Jin, Cheng et al. FAST TCP: Motivation, Architecture, Algorithms,
`Peiformance. Proceedings of IEEE Infocom 2004, pp. 1246-1259.
`Kaiserswerth, Matthias. The Parallel Protocol Engine. IEEE/ ACM
`Transactions on Networking, vol. 1, No.6, 1993. pp. 650-663.
`Kalmanek, Charles. A Retrospective View of ATM. ACM Computer
`Communication Review, vol. 32, No.5, 2002. pp. 13-19.
`Kamal, Humaira eta!. SCTP versus TCP for MPI. Proceedings of the
`2005 ACM/IEEE conference on Supercomputing, Seattle, Washing(cid:173)
`ton. 14 pages.
`Kanakia, Hemant eta!. The VMP Network Adapter Board (NAB):
`High-Performance Network Communication for Multiprocessors.
`ACM Computer Communication Review, vol. 18, No.4, 1988. pp.
`175-187.
`Kay, Jonathan eta!. The Importance of Non-Data Touching Process(cid:173)
`ing Overheads in TCPIIP. ACM Computer Communication Review,
`vol. 23, No.4, 1993. pp. 259-268.
`Kelly, Tom. Scalable TCP: Improving Performance in Highspeed
`Wide Area Networks. ACM Computer Communication Review, vol.
`33, No.2, 2003. pp. 83-91.
`Kent, Christopher A. et a!. Fragmentation Considered Harmful.
`ACM Computer Communication Review, vol. 17, No. 5, 1987. pp.
`390-401.
`Kermani, Parviz et a!. Virtual cut-through: A new computer com(cid:173)
`munciation switching technique. Computer Networks, vol. 3, No.4,
`1979. pp. 267-286.
`Kleinpaste, Karl eta!. Software Support for Outboard Buffering and
`Checksumming. ACM Computer Communication Review, vol. 25,
`No.4, 1995. pp. 87-98.
`Kline, Charley. Supercomputers on the Internet: A Case Study. ACM
`Computer Communication Review, vol. 17, No.5, 1987. pp. 27-33.
`Kuo, F.F. The Aloha System. ACM Computer Communication
`Review, vol. 4 No. 1, 1974. pp. 5-8.
`Lazowska, Edward D. Computing Research: A Looming Crisis. ACM
`Computer Communication Review, vol. 35, No.2, 2005. pp. 65-68.
`Leland, Will E. eta!. On the Self-Similar Nature of Ethernet Traffic.
`ACM Computer Communication Review, vol. 23, No. 4, 1993. pp.
`85-95.
`Leslie, Ian eta!. Fairisle: An ATM Network for the Local Area. ACM
`Computer Communication Review, vol. 21, No.4, 1991. pp. 327-
`336.
`Leslie, Ian M. eta!. The Architecture of the Universe Network. ACM
`Computer Communication Review, vol. 14, No.2, 1984. pp. 2-9.
`
`Leslie, Benet a!. User-level device drivers: Achieved performance. J.
`Comput. Sci. & Techno!., vol. 20, Sep. 2005. pp. 1-17.
`Lin, Mengjou et a!. Performance of High-Speed Network I/0 Sub(cid:173)
`systems: Case Study of A Fibre Channel Network. Proceedings of the
`1994 conference on Supercomputing, Washington D.C. pp. 17 4-183.
`Liu, Jinxing eta!. Performance Comparison ofMPI Implementations
`over InfiniBand, Myrinet and Quadrics. Proceedings of the 2003
`ACM/IEEE conference on Supercomputing, Phoenix, Arizona. pp.
`1-14.
`Lowekamp, Bruce et a!. Topology Discovery for Large Ethernet
`Networks. ACM Computer Communication Review, vol. 31, No.4,
`2001. pp. 237-248.
`Maeda, Chris et al. Protocol Service Decomposition for High-Per(cid:173)
`formance Networking. ACM Operating Systems Review, vol. 27,
`Issue 5, 1993. pp. 244-255.
`Mansley, Kieran et a!. Getting 10 Gb/s from Xen: Safe and Fast
`Device Access from Unprivileged Domains. Euro-Par Conference
`2007, Rennes, France. pp. 224-233.
`McAuley, Derek. A case for Virtual Channel Processors. Proceed(cid:173)
`ings at the ACM SIGCOMM 2003 Workshops. pp. 237-242.
`McAuley, Derek Robert. Protocol Design for High Speed Networks.
`PhD Thesis, University of Cambridge, 1989. 104 pages.
`McKenney, Paul E. eta!. Efficient Demultiplexing of Incoming TCP
`Packets. ACM Computer Communication Review, vol. 22, No. 4,
`1992. pp. 269-279.
`McQuillan, John M. eta!. An Overview of the New Routing Algorithm
`for the ARPANET. Proceedings of the 6th Data Communications
`Symposium, 1979. pp. 54-60.
`Metcalfe, Robert M. Ethernet: distributed packet switching for local
`computer networks. Communications of the ACM, vol. 19, Issue 7,
`1976. pp. 395-404.
`Michel, Jeffrey R. The Design and Evaluation of an Off Host Com(cid:173)
`munications Protocol Architecture. MSci Thesis, University of Vir(cid:173)
`ginia, 1993. 139 pages.
`Minshall, Greg et al. Flow labelled IP over ATM: design and ratio(cid:173)
`nale. ACM Computer Communication Review, vol. 36, No.3, 2006.
`pp. 79-92.
`Mockapetris, Paul V. et al. Development of the Domain Name System.
`ACM Computer Communication Review, vol. 18, No.4, 1988. pp.
`123-133
`Mogul, Jeffrey C. The Case for Persistent-Connection HTTP. ACM
`Computer Communication Review, vol. 25, No. 4, 1995. pp. 299-
`313.
`Mogul, Jeff. Presentation: Thoughts about TCP 0./floading; May
`2003 HotOS paper title: TCP ojfload is a dumb idea whose time has
`come. Aug. 2006. 3 pages.
`Moore, Gordon E. Cramming more components onto integrated cir(cid:173)
`cuits. Electronics, vol. 38, No.8, 1965. pp. 114-117.
`MPI: A Message-Passing Interface Standard. Message-Passing
`Interface Forum, University of Tennessee, Knoxville, 1994. 236
`pages.
`Muir, Steve et al. Piglet: A Low-Intrusion Vertical Operating System.
`Technical Report MS-CIS-00-04, University of Pennsylvania, 2000.
`pp. 1-15.
`Nagle, John. Congestion Control in IPITCP Internetworks. ACM
`Computer Communication Review, vol. 14, No.4, 1984. pp. 11-17.
`Partridge, Craig et al. Performance of Checksums and CRCS over
`Real Data. ACM Computer Communication Review, vol. 25, No.4,
`1995. pp. 68-76.
`Pope, Steven eta!. 1 OGb/s Ethernet Performance and Retrospective.
`ACM Computer Communication Review, vol. 37, No.2, 2007. pp.
`89-92.
`Pope, S.L. et a!. Enhancing Distributed Systems with Low-Latency
`Networking. Parallel and Distributed Computing and Networks,
`Brisbane, Australia, 1998. pp. 1-12.
`Pratt, Ian et a!. Arsenic: A User-Accessible Gigabit Ethernet Inter(cid:173)
`face. Proceedings ofiEEE Infocom 2001, pp. 67-76.
`Rangarajan, Murali eta!. TCP Servers: 0./floading TCP Processing in
`Internet Servers. Design, Implementation, and Performance. Tech(cid:173)
`nical Report DCR-TR-481, Computer Science Department, Rutgers
`University, 2002. pp. 1-14.
`
`004
`
`

`
`US 8,645,558 B2
`Page 5
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Regnier, Greg eta!. ETA: Experience with an Intel X eon Processor as
`a packet Processing Engine. IEEE Micro, vol. 24, No. 1, 1994. pp.
`24-31.
`Regnier, Greg eta!. TCP Onloadingfor Data Center Servers. Com(cid:173)
`puter, IEEE Computer Society, vol. 37, No. 11, 2004. pp. 48-58.
`Romanow, A. eta!. The Dynamics ofTCP Traffic over ATM Networks.
`ACM Computer Communication Review, vol. 24, No. 4, 1994. pp.
`79-88.
`Ross, Robert eta!. A Case Study in Application I/0 on Linux Clusters.
`Proceedings of the 2001 ACM/IEEE conference on Supercomputing,
`Denver. pp. 1-17.
`Rutsche, Erich. The Architecture of Gb/s Multimedia Protocol
`Adapter. ACM Computer Communication Review, vol. 23, No. 3,
`1993. pp. 59-68.
`Salmon, John et a!. Scaling of Beowulf-class Distributed Systems.
`Proceedings of the 1998 ACM/IEEE conference on Supercomputing,
`Orlando, Florida. pp. 1-18.
`Sancho, Jose Carlos eta!.. QuantifYing the Potential Benefit of Over(cid:173)
`lapping Communication and Computation in Large-Scale Scientific
`Applications. Proceedings of the 2006 ACM/IEEE conference on
`Supercomputing, Tampa, Florida. 16 pages.
`Sarolahti, Pasi eta!. F-RTO: An Enhanced Recovery Algorithm for
`TCP Retransmission Timeouts. CM Computer Communication
`Review, vol. 33, No.2, 2003. pp. 51-63.
`Shivam, Piyush et al. EMP: Zero-copy OS-bypass NIC-driven
`Gigabit Ethernet Message Passing. Proceedings of the 2001 ACM/
`IEEE conference on Supercomputing, Denver. pp. 1-8.
`Simmons, Margaret L. et al. Performance Comparison of the Cray-2
`and CrayX-MP/416 Supercomputers. Proceedings of the 1988ACM/
`IEEE conference on Supercomputing, Orlando, Florida. pp. 288-295.
`Singh, Raj K. eta!. A Programmable HIP PI Interface for a Graphics
`Supercomputer. Proceedings of the 1993 ACM/IEEE conference on
`Supercomputing, Portland, Oregon. pp. 124-132.
`Singh, Raj K. et a!. A Programmable Network Interface for a Mes(cid:173)
`sage-Based Multicomputer. ACM Computer Communication
`Review, vol. 24, No.3, p. 8-17, 1994.
`Sistare, Steven J. et al. Ultra-High Performance Communication with
`MPI and the Sun Fire™ Link Interconnect. Proceedings of the 2002
`ACM/IEEE conference on Supercomputing, Baltimore. pp. 1-15.
`Smith, Jonathan M. et al. Giving Applications Access to Gb/s Net(cid:173)
`working. IEEE Network, vol. 7, Issue 4, 1993. pp. 44-52.
`Smith, Jonathan. The Influence of ATM on Operating Systems. ACM
`Computer Communication Review, vol. 32, No.5, 2002. pp. 29-37.
`Steenkiste, Peter. Analyzing Communication Latency using the Nec(cid:173)
`tar Communication Processor. ACM Computer Communication
`Review, vol. 22, No.4, 1992. pp. 199-209.
`Stone, Jonathan eta!. When the CRC and TCP Checksum Disagree.
`ACM Computer Communication Review, vol. 30, No. 4, 2000. pp.
`309-319.
`Sayantan, Sur et a!. High-Performance and Scalable MPI over
`InfiniBand with Reduced Memory Usage: An In-Depth Performance
`Analysis. Proceedings of the 2006 ACM/IEEE conference on
`Supercomputing, Tampa, Florida. 13 pages.
`Tennenhouse, D. L. et al. Towards an Active Network Architecture.
`ACM Computer Communication Review, vol. 26, No. 2, 1996. pp.
`15-20.
`Thacker, Charles P. eta!. Firefly: a Multiprocessor Workstation. ACM
`Operating Systems Review, vol. 21, Issue 4, 1987. pp. 164-172.
`Thekkath, Chandramohan A. et al. Implementing Network Protocols
`at User Level. ACM Computer Communication Review, vol. 23, No.
`4, 1993. pp. 64-73.
`Touch, Joe et al. Experiences with a Production Gigabit LAN. Gigabit
`Networking Workshop '97 Meeting, Kobe, Japan. 10 pages.
`Touch, Joe et al. Host-based Routing Using Peer DMA. Gigabit
`Networking Workshop '97 Meeting, Kobe, Japan. 2 pages.
`Traw, C. Brendan S. eta!. A High-Performance Host Interface for
`ATM Networks. ACM Computer Communication Review, vol. 21,
`No.4, 1991. pp. 317-325.
`
`Traw, C. Brendan S. et al. Hardware/Software organization of a high
`performanceATM host interface. IEEE Journal on Selected Areas in
`Communications, 1993. pp. 240-253.
`Tsudik, Gene. Message Authentication with One-Way Hash Func(cid:173)
`tions. ACM Computer Communication Review, vol. 22, No.5, 1992.
`pp. 29-38.
`Vis, Jan.A Simple LAN Peiformance Measure. ACM Computer Com(cid:173)
`munication Review, vol. 24, No. 1, 1994. pp. 7-11.
`Warren, Michael S. et al. Avalon: An Alpha/Linux Cluster Achieves
`10 Gfiopsfor $150k. Proceedings of the 1998 ACM/IEEE conference
`on Supercomputing, Orlando, Florida. pp. 1-10.
`Wassermann, Harvey J. et a!. Performance Evaluation of the SGI
`Origin2000: A Memory-Centric Characterization of LANL ASCI
`Applications. Proceedings of the 1997 ACM/IEEE conference on
`Supercomputing, San Jose, California. pp. 1-11.
`Wetherall, David. 10 Networking Papers: Readings for Protocol
`Design. ACM Computer Communication Review, vol. 36, No. 3,
`2006. pp. 77-78.
`Wilkes, Maurice v. eta!. The Cambridge Model Distributed System.
`ACM SIGOPS Operating Systems Review, vol. 14, Issue 1, 1980. pp.
`21-29.
`Wray, Stuart eta!. The Medusa Applications Environment. Proceed(cid:173)
`ings of the International Conference on Multimedia Computing and
`Systems, Boston, 1994. pp. 265-273.
`Zelkowitz, Marvin. Interrupt Driven Programming. Communica(cid:173)
`tions oftheACM, vol. 14, No.6, 1971. pp. 417-418.
`Bush, R. eta!. Some Internet Architectural Guidelines and Philoso(cid:173)
`phy. The Internet Society, Dec. 2002, pp. 1-28.
`Dixon, Michael J. System support for multi-service traffic. University
`of Cambridge Computer Laboratory Technical Report No. 245, Jan.
`1992, 113 pages.
`Feng, Wu-chun, eta!. Performance Characterization of a 1 0-Gigabit
`Ethernet TOE. Proceedings of the 13th Symposium on High Perfor(cid:173)
`mance Interconnects, 2005, pp. 1-6.
`Hrvoye, Bilic eta!. Deferred Segmentation for Wire-Speed Transmis(cid:173)
`sion of Large TCP Frames over Standard GbE Networks. Proceed(cid:173)
`ings of the 9th Symposium on High Performance Interconnects,
`2001, 5 pages.
`Hrvoye, Bilic et al. Presentation of: Deferred Segmentation for Wire(cid:173)
`Speed Transmission of Large TCP Frames over Standard GbE Net(cid:173)
`work, given at HOTI' 01 9th Symposium on High Performance
`Interconnects, 2001, 9 pages.
`Hurwitz, Justin eta!. Initial End-to-End Performance Evaluation of
`1 0-Gigabit Ethernet, Proceedings of IEEE Hot Interconnects: 11th
`Symposium on High Performance Interconnects, Aug. 2003, 6 pages.
`Makineni, Srihari et a!. Architectural Characterization of TCPIIP
`Packet Processing on the Pentium® M Processor. Proceedings of the
`lOth International Symposium on High Performance Computer
`Architecture, 2004, 11 pages.
`Mogul, Jeffrey C. TCP ojfload is a dumb idea whose time has come.
`Proceedings of HotOS IX: The 9th Workshop on Hot Topics in
`Operating Systems, May 18-21, 2003, pp. 25-30.
`Montry, Gary. From Presentation: The NE01 0 iWARP Adapter. Open
`Fabrics Alliance at http:/ /www.openfabrics.org, 2006, 8 pages.
`Partridge, Craig. How Slow Is One Gigabit Per Second? ACM Com(cid:173)
`puter Communication Review, vol. 20, No. 11990, Jun. 5, 1989, pp.
`44-53.
`Petrini, Fabrizio. Presentation of: Protocol Offloading vs On-load(cid:173)
`ing in High-Peiformance Networks. 141
`h IEEE Symposium on High(cid:173)
`Performance Interconnects, Aug. 2006, 4 pages.
`Regnier, Greg. Presentation: Protocol Onload vs. Ojfload, May 2006,
`Intel. 1 page.
`Rutsche, Erich eta!. TCPIIP on the Parallel Protocol Engine. Pro(cid:173)
`ceedings of the IFIP TC6/WG6.4 Fourth International Conference on
`High Performance Networking IV, 1993, pp. 119-134.
`Sterling, Thomas eta!. Beowulf: A Parallel Workstation for Scientific
`Computation. Proceedings of the 24th International Conference on
`Parallel Processing, 1995, 4 pages.
`* cited by examiner
`
`Primary Examiner- Jeong S Park
`
`005
`
`

`
`US 8,645,558 B2
`Page 6
`
`(74) Attorney, Agent, or Firm- Fish & Richardson P.C.
`
`ABSTRACT
`(57)
`A data processing system arranged for receiving over a net(cid:173)
`work, according to a data transfer protocol, data directed to
`any of a plurality of destination identities, the data processing
`system comprising: data storage (event queues 31-33) for
`storing data received over the network; and a first processing
`arrangement for performing processing in accordance with
`the data transfer protocol on received data in the data storage,
`for making the received data available to respective destina(ci