`
`United States 
`Patent File History 
`
`Tab Listings 
`A. References (if applicable) 
`A1‐U.S. References 
`A2‐Foreign References 
`B. Jacket (face of file, contents flap, index of 
`claims, PTO 270, searched) 
`C.Printed Patent 
`D. Specification (serial no. Sheet, abstract, 
`specification, claims) 
`E.Oath 
`E1‐Small Entity Status (if applicable) 
`F.Drawing Figures (if applicable) 
`G. USPTO / Applicant Correspondence 
`H. Original Patent Application (in cases of 
`FWC) 
`
` 
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 1 of 426
`
`

`

` 
`
`The Publications are found As Is 
`
` 
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 2 of 426
`
`

`

`AUG l 0 200~
`
`U.S. UTILITY Patent Application
`
`~;, APPLNUMA djlllNG DATE
`'l 0210112003
`10351,'s::w
`
`1
`
`/
`
`. ,
`
`I
`o ,( ,/'117 w--
`'t co N T1 N v1 NG DP..";"J\ VERP::C:.0:
`This aµpl:cation is a CiP of 1Oi03•i,197 12/28/20'CJ'1
`which claims banefit of 60/259,269 12/29/2000
`This application 10/361,837
`claims benefit •)f 60/355,509 02/0f.12002
`
`I'
`I!
`
`I
`
`i ~' .
`
`i·
`
`I
`'-"'\ t..-
`!Ur-
`, '"' t-=OREIGN APPLiCATIONS VERIFIED:
`I
`ii
`~ --~r,,;w:,;,.,-..=--.-""'zwu• ,,. .. ,
`PG-Pl.!.J DO NOT PUtiLISH c
`
`fi
`--- -; .·~·~. -~.._., ·:~-~:::,77,;...,, ·.;i·•-:;;;;:;;;;--;:;;;;;;;;:.-:;;;::;;.--:---;;w;:,-.;:::~.-:-~
`. I
`l
`~ESCIN~ D
`
`Fo;eign priority claimed
`:'.'.5 USC 119 ccnditions met
`Verified <.nd Acl<oio:v!edged Examiner:;'s intiah:;
`
`CJ yes}? no
`CJ yes ;sr· no
`/'/1 lfl
`.
`ITLE : Tcols and techniques for directing packets over disparate networks
`'
`.
`,__ - - - - - - - - - - - - - - - - - - - - - - -__ ; : ; , ; ; ;U .S .DE ?T . OP COMM.IPAT . .3. Tl!'-PT0-~3Cl(R'V. 12:2.:!,L
`
`·rTTORNEY DOCKET NO
`
`13003.2. 11A
`
`7//J/61
`
`NOTICE OF ALLOWANCE MAILED
`
`ISSUE FEE
`Data Paid
`Amount Dua
`
`/
`/'
`.tl iftt~
`I~ Ii /04 .. ~M
`D TERMINAL
`Ssl~[ C(C,~~Pi:
`UL I L ~~ ail[,
`
`/ .
`
`CLAIMS ALLOWED
`
`Total Claims
`() (.,
`
`Print Cl1lm for
`O.G
`/
`
`DRAWING
`Flga.Drwg.
`JI
`
`Print Fig.
`b
`
`MELVIN MARCELO
`PRIMARY EXAMINER
`Primary Examiner
`
`PREPARED FOR ISSUE -
`erein may be restricted.
`WARNING: The information disclo
`· Unauthorized disclosure may be prohibited by the United States Code Title 35,
`Sections 122, 181 and 368, Possession outside the U.S. Patent & Trademark
`Office is restricted to authorized e
`lo ees and contractors onl .
`FILED WITH: D DISK (CRF)
`
`D CD-ROM
`
`(Attached In pocket on right Inside ftap)
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 3 of 426
`
`

`

`I
`
`•
`
`-.-.....,,
`
`'
`
`>
`
`.. ,
`
`,
`
`J
`
`~WCO ...... O>WO_..
`
`I 11111111111111111111111111111111111111111 Ill\ Ill\
`10361837
`
`'
`
`. . .
`10/361837
`
`~
`
`••
`
`I
`
`'
`
`J
`
`•
`
`.
`
`.
`
`~
`
`. "
`
`11111111111111111111111111111111 llll llll
`02/07/03
`
`CONTENTS
`Date
`Received
`(Incl. C. of M.)
`or
`Date Mailed
`
`Date
`Received
`(Incl. C. of M.)
`or
`Date Majied
`
`Appl/cation
`
`(o
`
`.. papers.
`
`1.
`
`2. XDs.
`3. ·~ (U_\ ~':'. ~ ,ttlflb)
`
`·ill:.
`31. ______ •
`
`~) rz,J~ 32. _····_· .... _~ _
`
`.. ~_ ... __ ·! _
`
`33. _____ _
`
`14.
`
`15.
`
`16,
`
`17.
`
`18.
`
`19.
`
`20.
`
`21.
`
`22.
`
`23.
`
`24.
`
`25.
`
`28.
`
`27.
`
`28.
`
`29.
`
`30.
`
`44.
`
`45.
`
`46.
`
`47.
`
`48.
`
`48.
`
`50.
`
`51.
`
`52.
`
`53.
`
`54.
`
`55.
`
`58.
`
`57.
`
`58.
`
`59.
`
`60.
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 4 of 426
`
`

`

`ORIGINAL
`GLASS
`SUBCLASS
`'2 ·'f-o
`'J ·? <;!
`7·
`0 7
`
`ISSUE SLIP STAPLE AREA (for additional cross-references)
`ISSUING CLASSIFICATION
`CROSS REFERENCE(S
`SUBCLASS (ONE SUBCLASS PER BLOCK)
`d. i; ;l 39J
`
`CLASS
`3 -.:re
`
`INTERNATIONAL
`CLASSIFICATION
`
`I-} 0 iy L,._
`
`,.,,.,.
`
`I (:; L/
`I
`I
`I
`I
`
`/
`
`~ :g
`
`i!
`u::
`
`I
`
`I /
`
`'--
`
`I'
`12 v
`a v
`v
`I a.5 0
`1. 6 In --
`> 7 " ._-
`
`~/
`
`lo'
`
`·-
`
`-;:;
`
`/\ Continued on Issue Slip Inside File Jacket
`INDEX OF CLAIMS
`- (Through numeral) .•• canceled
`./ ............... Rejected
`N .......•...•.•• Non-elected A .•••..•........ Appeal
`...............
`=
`···············
`............. Resbicted
`Allowed
`I ............... lnte rference 0 ............... Oblected
`+
`lelaim
`Date
`Claim
`Date
`Claim
`Date
`0, IW
`!
`~
`1J'; I~
`ii ~
`ii ·~
`ry ~
`c
`~ 0
`LC 0
`101
`51
`102
`52
`103
`53
`104
`54
`105
`55
`106
`56
`107
`57
`108
`58
`109
`59
`110
`60
`111
`61
`112
`62
`113
`63
`114
`64.
`115
`65
`116
`66
`117
`67
`118
`68
`119
`69
`120
`70
`121
`71
`122
`72
`123
`73
`124
`74
`125
`75
`126
`76
`127
`71
`128
`78
`129
`79
`130
`80
`131
`81
`132
`82
`133
`83
`134
`84
`135
`85
`136
`86
`137
`87
`138
`88
`139
`89
`140
`90
`141
`91
`142
`92
`143
`93
`144
`94
`95
`145
`146
`96
`147
`97
`148
`98
`149
`99
`150
`100
`..
`If more than 150 claims or 9 actions staple add1t1onal sheet here
`
`~-
`~ v
`1,j
`Ill
`14
`. ~
`~ .t
`~ 'tl
`::r 14
`<)( 15
`v1 16
`IO 17
`ii 18
`J// 19
`(} 20
`1'-1 21
`l< 22 :::::
`( ~ 23 v
`ii r- 24 v
`
`\~6 25 v --
`
`-~.//
`•Cf 2~ n
`28. v
`29,,;.
`
`c
`
`' -::: -
`
`' i
`
`IJ'll
`i.J
`
`1 i;~/l
`
`.::
`•• v
`3. ~ :::
`-:::
`J..?•. 3'
`::;
`' '}~ 35 :::
`,:;;
`36
`37
`38
`39
`40
`41
`42
`43
`44
`45
`46
`47
`48
`49
`50
`
`;:::
`
`ij
`~
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 5 of 426
`
`

`

`NAME
`
`IDNO.
`
`DATE
`
`POSITION
`=EE DETERMINATION
`:1LE ASSEMBLY
`lUALITY CHECK
`)CANNING
`'.:LASSIFIER
`=ORMALITY REVIEW
`~ESPONSE
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 6 of 426
`
`

`

`SEARCH NOTES
`(List databases searched. Attach
`search strateav inside.)
`Date Exmr.
`
`SEARCH
`
`'
`
`'
`
`Class Sub. Date Exmr.
`;). S' ;)-. )
`?~ 3~/-.
`~ 7. 0
`;. J
`13 .s
`>;,ct I
`
`I 0.1-< ( O· iV1f ?11 t-v---
`
`l
`
`...
`
`t~ (1,~41-.{
`
`Q).-J. )· J-ro Y 0'\ /l'V
`..
`
`l/'
`
`l' /I
`I
`
`..
`INTERFERENCE,SEARCHED
`iDate Exmr.
`Class Sub.
`I
`J-:7~ ~ ?
`
`., ''.)..<>
`'7'
`I
`
`~~;; 7
`
`v~.ici-0oy ...., ,._
`)
`
`I
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 7 of 426
`
`

`

`(12) United States Patent
`Datta et al.
`
`I lllll llllllll Ill lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US006775235B2
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,775,235 B2
`Aug. 10, 2004
`
`(54) TOOLS AND TECHNIQUES FOR
`DIRECTING PACKETS OVER DISPARATE
`NETWORKS
`
`(75)
`
`Inventors: Sanchaita Datta, Salt Lake City, UT
`(US); Ragula Bhaskar, Salt Lake City,
`UT (US)
`
`(73) Assignee: Ragula Systems, Salt Lake City, UT
`(US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 10/361,837
`
`(22) Filed:
`
`Feb. 7, 2003
`
`(65)
`
`Prior Publication Data
`
`US 2003/0147408 Al Aug. 7, 2003
`
`................. 709/240
`9/1999 Kitai et al.
`5,948,069 A
`6,016,307 A * 1/2000 Kaplan et al.
`.............. 370/238
`6,119,170 A * 9/2000 Schoffelman et al. ....... 709/244
`6,128,298 A * 10/2000 Wootton et al.
`............ 370/392
`6,253,247 Bl
`6/2001 Bhaskar et al. ............. 709/237
`6,295,276 Bl
`9/2001 Datta et al. ................. 370/218
`6,339,595 Bl
`1/2002 Rekhter et al.
`............. 370/392
`6,438,100 Bl
`8/2002 Halpern et al. ............. 370/218
`6,449,259 Bl
`9/2002 Allain et al. ................ 370/253
`6,456,594 Bl
`9/2002 Kaplan et al.
`.............. 370/238
`6,493,341 Bl
`12/2002 Datta et al. ................. 370/392
`6,493,349 Bl
`12/2002 Casey ........................ 370/409
`6,665,702 Bl * 12/2003 Zisapel et al. .............. 718/105
`
`OTHER PUBLICATIONS
`
`'Radware announces LinkProof: The first IP Load Balancing
`Solution for networks with multiple ISP connection', Press
`Release, published Oct. 7, 1999. *
`'Radware Balances the Network', Internet Traffic Manage(cid:173)
`ment Center, published Jan. 1, 2000. *
`'Global Product Spotlight: Radware Linkproof', Network(cid:173)
`Magazine.com, published Dec. 1, 1999. *
`
`Related U.S. Application Data
`
`(List continued on next page.)
`
`(63)
`
`(60)
`
`(51)
`(52)
`(58)
`
`(56)
`
`Continuation-in-part of application No. 10/034,197, filed on
`Dec. 28, 2001.
`Provisional application No. 60/355,509, filed on Feb. 8,
`2002, and provisional application No. 60/259,269, filed on
`Dec. 29, 2000.
`
`Int. Cl.7 ................................................ H04L 12/64
`U.S. Cl. ........................ 370/238; 370/252; 370/352
`Field of Search ................................. 370/252, 352,
`370/230, 235, 238
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,398,012 A
`5,420,862 A
`5,473,599 A
`5,737,526 A
`5,898,673 A
`
`3/1995 Derby et al. ........... 340/825.03
`5/1995 Perlman .................. 370/85.13
`....................... 370/16
`12/1995 Li et al.
`4/1998 Periasamy et al. . ... . 395/200.06
`4/1999 Riggan et al.
`.............. 370/237
`
`Primary Examiner-Melvin Marcelo
`(74) Attorney, Agent, or Firm-Thorpe North & Western
`LLP
`
`(57)
`
`ABSTRACT
`
`Methods, configured storage media, and systems are pro(cid:173)
`vided for communications using two or more disparate
`networks in parallel to provide load balancing across net(cid:173)
`work connections, greater reliability, and/or increased secu(cid:173)
`rity. A controller provides access to two or more disparate
`networks in parallel, through direct or indirect network
`interfaces. When one attached network fails, the failure is
`sensed by the controller and traffic is routed through one or
`more other disparate networks. When all attached disparate
`networks are operating, one controller preferably balances
`the load between them.
`
`24 Claims, 6 Drawing Sheets
`
`INTERNET 500
`
`LINE 4
`
`ROUTER
`104
`
`/LINE 5
`
`ROUTER
`105
`
`LINE 7
`
`ROUTER
`105
`
`FRAME RELAY I POINT-TO-POINT NETWORK 106/204
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 8 of 426
`
`

`

`US 6,775,235 B2
`Page 2
`
`OIBER PUBLICATIONS
`
`'Radware Seeks Solutions to Easy-Access Problems', South
`China Morning Post, published Dec. 7, 1999.*
`B. Gleeson et al., "A Framework for IP Based Virtual Private
`Networks," RFC 2764 (Feb. 2000).
`U.S. patent application, Attorney Docket No. 3003.2.9A; see
`USPTO published application No. US-2002--0087724--Al,
`Jul. 4, 2002.
`T. Liao et al., "Using multiple links to interconnect LANs
`and public circuit switched data networks," Proc. Int. Con(cid:173)
`ference on Communications Systems: Towards Global Inte(cid:173)
`gration, vol. 1, Singapore, 59 Nov. 1990, pp. 289-293.
`Press release from www.coyotepoint,com, Sep. 8, 1997.
`Network Address Translation Technical Discussion, from
`safety.net; no later than May 7, 1999.
`Higginson et al., "Development of Router Clusters to Pro(cid:173)
`vide Fast Failover in IP Networks," from www.asia-paci(cid:173)
`fic.digital.com; no later than Sep. 29, 1998.
`
`Pages from www.navpoint.com; no later than Dec. 24, 2001.
`"The Basic Guide to Frame Relay Networking", pp. 1-85,
`copyright date 1998.
`"NNI & UNI", pp. 1-2, Nov. 16, 2001.
`"Disaster Recovery for Frame Relay Networks", pp. 1-14,
`no later than Dec. 7, 2001.
`T. Nolle, "Watching Your Back", pp. 1-3, Nov. 1, 1999.
`"Multi-Attached and Multi-Homed Dedicated Access", pp.
`1-5, no later than Dec. 8, 2001.
`Feibel, "Internetwork Link," Novell's® Complete Encyclo(cid:173)
`pedia of Networking, copyright date 1995.
`Tanenbaum, Computer Networks (3rd Ed.), pp. 396-406;
`copyright date 1996.
`Wexler, "Frame Relay and IPVPNs: Compete Or Coexist?",
`from www.bcr.com; Jul. 1999.
`
`* cited by examiner
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 9 of 426
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 1 of 6
`
`US 6,775,235 B2
`
`ROUTER A1
`105
`
`SITE 1
`102
`
`I
`FRAME RELAY
`NETWORK A
`106
`
`ROUTER 81
`105
`I
`FRAME RELAY
`NETWORK B
`108
`
`I
`ROUTERA2
`105
`
`SITE2
`102
`
`I
`ROUTER 82
`105
`
`(PRIOR ART)
`Fig. 1
`
`SITE 1
`102
`
`ROUTER 1 105
`
`FAILOVER
`COMPONENT 202
`
`I
`FRAME RELAY
`NETWORK 106
`
`I
`ISDN NETWORK
`LINK 204
`
`I
`
`l
`ROUTER 2 105
`
`FAILOVER
`COMPONENT 202
`
`(PRIOR ART)
`Fig. 2
`
`SITE 2
`102
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 10 of 426
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 2 of 6
`
`US 6,775,235 B2
`
`CORPORATION OR OTHER ENTITY 302
`I SITE 41
`102
`I
`FRAME RELAY
`NETWORK B
`1.Q§
`
`[SiTEil
`lJgLJ
`I
`FRAME RELAY
`NETWORK A
`1.Q§
`
`~ \s~~o~31 ~ 1s~~D~61
`
`SITE 1
`102
`
`(PRIOR ART)
`Fig. 3
`
`ROUTER 1
`105
`I
`FRAME RELAY
`NETWORK A 106
`
`I
`NETWORK-TO-NETWORK
`INTERFACE 402
`
`I
`FRAME RELAY
`NETWORK B 106
`
`I
`ROUTER 2
`105
`
`-----fSiTE2l
`SIT
`lJgLJ
`
`(PRIOR ART)
`Fig. 4
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 11 of 426
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 3 of 6
`
`US 6,775,235 B2
`
`ROUTER A1
`105
`
`SITE 1
`102
`
`ROUTER B1
`104
`
`I
`
`FRAME RELAY
`NETWORK A
`106
`
`I
`INTERNET I VIRTUAL
`PRIVATE NETWORK
`500 / 502
`
`I
`ROUTER A2
`105
`
`SITE2
`102
`
`I
`ROUTER B2
`104
`
`(PRIOR ART)
`Fig. 5
`
`SITE
`102
`
`-
`
`MULTIPLE DISPARATE NETWORK ACCESS
`CONTROLLER 602
`
`SITE INTERFACE 702
`PACKET PATH SELECTOR (E.G., LOAD
`BALANCING, REDUNDANCY, SECURITY) 704
`
`INTERFACE
`706
`
`INTERFACE
`706
`
`INTERFACE
`706
`
`TOA
`NETWORK
`BY PATH
`A1
`
`1r
`
`1r
`
`TOA
`NETWORK
`BY PATH
`A2
`
`,,.
`
`TO
`A
`ORK
`NETW
`TH
`BYPA
`A3
`
`Fig. 7
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 12 of 426
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 4 of 6
`
`US 6,775,235 B2
`
`I
`
`INTERNET 500
`
`I
`
`LINE 1 ;
`
`LINE 2
`
`;
`
`LINE 3 /
`
`I
`
`LINE4
`
`ROUTER ROUTER
`104
`104
`
`ROUTER ROUTER
`104
`104
`
`Cl)
`z
`0
`I-
`()
`w
`z
`z
`0
`()
`I-w
`z
`0::::
`w
`I-
`z -
`SITEA CONTROLLER ~ CONTROLLER
`102 -
`
`VPN
`604
`
`VPN
`604
`
`602
`
`VPN
`604
`
`I
`
`,_
`
`602
`
`SITE C
`102
`
`I
`
`LINE 5
`
`LINE 6 \
`
`LINE 7 \
`
`ROUTER
`105
`
`ROUTER
`105
`
`ROUTER
`105
`
`I
`I
`I
`FRAME RELAY I POINT-TO-POINT NETWORK 106/204
`
`0::::
`0 w Cl)
`>- z z
`<( - 0
`_J _J I-
`w 0 ()
`o::::ww
`W Cl)Z
`:2 <Cz
`<( ~ 0
`0::::
`()
`LL
`
`Fig. 6
`
`INTERNET 500
`
`I
`
`I
`ROUTER X
`104
`
`I
`SITEA CONTROLLER
`102 -
`A602
`I
`ROUTER Y
`105
`
`I
`ROUTER Z
`104
`I
`SITE B
`CONTROLLER
`B 602 ~ 102
`
`I
`ROUTER W
`105
`
`I
`
`I
`I
`FRAME RELAY NETWORK 106
`Fig. 10
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 13 of 426
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 5 of 6
`
`US 6,775,235 B2
`
`SPECIFY PATH SELECTOR CRITERIA 800
`
`SEND PACKET(S) TO CONTROLLER 802
`
`DETECT NETWORK FAILURE 804
`
`ROUTE AROUND FAILURE 806
`
`Fig. 8
`
`i
`OBTAIN ADDRESS
`RANGE
`INFORMATION 900
`i
`
`~
`
`.....
`
`i
`.. OBTAIN SYSTEM
`...
`TOPOLOGY
`INFORMATION 902
`i
`
`.. I ...
`I •
`LOOK FOR ADDRESS TO "KNOWN" DESTINATION 906
`i
`SELECT PATH TO A DISPARATE NETWORK 908
`USE LOAD BALANCING CRITERION 910
`
`RECEIVE PACKET FROM LOCAL SITE 904
`
`USE CONNECTIVITY CRITERION 912
`
`USE SECURITY CRITERION 914
`
`I
`
`I
`
`I
`
`i
`i
`
`MODIFY PACKET DESTINATION ADDRESS 916 I
`
`FORWARD PACKET ON SELECTED PATH 918
`I ...
`Fig. 9
`
`I
`
`I
`
`I
`
`1...--
`1ir
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 14 of 426
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 6 of 6
`
`US 6,775,235 B2
`
`I
`
`INTERNET 500
`
`I
`
`I
`I
`ROUTER Z
`ROUTER X
`104
`104
`I
`I
`CONTROLLER CONTROLLER
`-
`B 602
`A602
`I
`I
`ROUTER W
`ROUTER Y
`105
`105
`I
`I
`FRAME RELAY NETWORK 106
`Fig. 11
`
`VPNA
`101 -
`
`I
`SITEA
`102
`
`I
`
`VPN B
`101
`
`I
`SITE B
`102
`
`I
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 15 of 426
`
`

`

`US 6,775,235 B2
`
`1
`TOOLS AND TECHNIQUES FOR
`DIRECTING PACKETS OVER DISPARATE
`NETWORKS
`
`RELATED APPLICATIONS
`
`This application claims priority to commonly owned
`copending U.S. provisional patent application serial No.
`60/355,509 filed Feb. 8, 2002, which is also incorporated
`herein by reference. This application is a continuation-in(cid:173)
`part of U.S. patent application Ser. No. 10/034,197 filed
`Dec. 28, 2001, which claims priority to U.S. provisional
`patent application serial No. 60/259,269 filed Dec. 29, 2000,
`each of which is also incorporated herein by reference.
`
`FIELD OF THE INVENTION
`
`The present invention relates to computer network data
`transmission, and more particularly relates to tools and
`techniques for communications using disparate parallel
`networks, such as a virtual private network ("VPN") or the
`Internet in parallel with a point-to-point, leased line, or
`frame relay network, in order to help provide benefits such
`as load balancing across network connections, greater
`reliability, and increased security.
`
`TECHNICAL BACKGROUND OF THE
`INVENTION
`
`2
`connection. Frame relays are used as examples throughout
`this document, but the teachings will also be understood in
`the context of point-to-point networks.
`A frame relay or point-to-point network may become
`suddenly unavailable for use. For instance, both MCI World(cid:173)
`Com and AT&T users have lost access to their respective
`frame relay networks during major outages. During each
`outage, the entire network failed. Loss of a particular line or
`node in a network is relatively easy to work around. But loss
`10 of an entire network creates much larger problems.
`Tools and techniques to permit continued data transmis(cid:173)
`sion after loss of an entire frame relay network that would
`normally carry data are discussed in U.S. patent application
`Ser. No. 10/034,197 filed Dec. 28, 2001 and incorporated
`15 herein. The '197 application focuses on architectures involv(cid:173)
`ing two or more "private" networks in parallel, whereas the
`present application focuses on architectures involving dis(cid:173)
`parate networks in parallel, such as a proprietary frame relay
`network and the Internet. Note that the term "private net-
`20 work" is used herein in a manner consistent with its use in
`the '197 application (which comprises frame relay and
`point-to-point networks), except that a "virtual private net(cid:173)
`work" as discussed herein is not a "private network". Virtual
`private networks are Internet-based, and hence disparate
`25 from private networks, i.e., from frame relay and point-to-
`point networks. To reduce the risk of confusion that might
`arise from misunderstanding "private network" to comprise
`"virtual private network" herein, virtual private networks
`will be henceforth referred to as VPNs. Other differences
`30 and similarities between the present application and the '197
`application will also be apparent to those of skill in the art
`on reading the two applications.
`Various architectures involving multiple networks are
`known in the art. For instance, FIG. 1 illustrates prior art
`configurations involving two frame relay networks for
`increased reliability; similar configurations involve one or
`more point-to-point network connections. Two sites 102
`transmit data to each other (alternately, one site might be
`40 only a data source, while the other is only a data destination).
`Each site has two border routers 105. Two frame relay
`networks 106, 108 are available to the sites 102 through the
`routers 105. The two frame relay networks 106, 108 have
`been given separate numbers in the figure, even though each
`is a frame relay network, to emphasize the incompatibility of
`frame relay networks provided by different carriers. An
`AT&T frame relay network, for instance, is incompatible(cid:173)
`in details such as maximum frame size or switching
`capacity-with an MCI WorldCom frame relay network,
`even though they are similar when one takes the broader
`view that encompasses disparate networks like those dis-
`cussed herein. The two frame relay providers have to agree
`upon information rates, switching capacities, frame sizes,
`etc. before the two networks can communicate directly with
`each other.
`A configuration like that shown in FIG. 1 may be actively
`and routinely using both frame relay networks A and B. For
`instance, a local area network (LAN) at site 1 may be set up
`to send all traffic from the accounting and sales departments
`60 to router Al and send all traffic from the engineering
`department to router Bl. This may provide a very rough
`balance of the traffic load between the routers, but it does not
`attempt to balance router loads dynamically in response to
`actual traffic and thus is not "load-balancing" as that term is
`65 used herein.
`Alternatively, one of the frame relay networks may be a
`backup which is used only when the other frame relay
`
`Organizations have used frame relay networks and point(cid:173)
`to-point leased line networks for interconnecting geographi(cid:173)
`cally dispersed offices or locations. These networks have
`been implemented in the past and are currently in use for
`interoffice communication, data exchange and file sharing.
`Such networks have advantages, some of which are noted
`below. But these networks also tend to be expensive, and 35
`there are relatively few options for reliability and redun(cid:173)
`dancy. As networked data communication becomes critical
`to the day-to-day operation and functioning of an
`organization, the need for lower cost alternatives for redun(cid:173)
`dant back-up for wide area networks becomes important.
`Frame relay networking technology offers relatively high
`throughput and reliability. Data is sent in variable length
`frames, which are a type of packet. Each frame has an
`address that the frame relay network uses to determine the
`frame's destination. The frames travel to their destination 45
`through a series of switches in the frame relay network,
`which is sometimes called a network "cloud"; frame relay is
`an example of packet-switched networking technology. The
`transmission lines in the frame relay cloud must be essen(cid:173)
`tially error-free for frame relay to perform well, although 50
`error handling by other mechanisms at the data source and
`destination can compensate to some extent for lower line
`reliability. Frame relay and/or point-to-point network ser(cid:173)
`vices are provided or have been provided by various carriers,
`such as AT&T, Qwest, XO, and MCI WorldCom.
`Frame relay networks are an example of a network that is
`"disparate" from the Internet and from Internet-based virtual
`private networks for purposes of the present invention.
`Another example of such a "disparate" network is a point(cid:173)
`to-point network, such as a Tl or T3 connection. Although
`the underlying technologies differ somewhat, for purposes
`of the present invention frame relay networks and point-to(cid:173)
`point networks are generally equivalent in important ways,
`such as the conventional reliance on manual switchovers
`when traffic must be redirected after a connection fails, and
`their implementation distinct from the Internet. A frame
`relay permanent virtual circuit is a virtual point-to-point
`
`55
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 16 of 426
`
`

`

`US 6,775,235 B2
`
`4
`in parallel would probably not consider NNis pertinent,
`because they were used for serial configurations rather than
`parallel ones, and adding networks in a serial manner does
`not improve reliability.
`Internet-based communication solutions such as VPNs
`and Secure Sockets Layer (SSL) offer alternatives to frame
`relay 106 and point-to-point leased line networks such as
`those using an ISDN link 204. These Internet-based solu(cid:173)
`tions are advantageous in the flexibility and choice they offer
`10 in cost, in service providers, and in vendors. Accordingly,
`some organizations have a frame relay 106 or leased line
`connection (a.k.a. point-to-point) for intranet communica(cid:173)
`tion and also have a connection for accessing the Internet
`500, using an architecture such as that shown in FIG. 5.
`But better tools and techniques are needed for use in
`architectures such as that shown in FIG. 5. In particular,
`prior approaches for selecting which network to use for
`which packet(s) are coarse. For instance, all packets from
`department X might be sent over the frame relay connection
`20 106 while all packets from department Y are sent over the
`Internet 500. Or the architecture might send all traffic over
`the frame relay network unless that network fails, and then
`be manually reconfigured to send all traffic over a VPN 502.
`Organizations are still looking for better ways to use
`Internet-based redundant connections to backup the primary
`frame relay networks. Also, organizations wanting to change
`from frame relay and point-to-point solutions to Internet(cid:173)
`based solutions have not had the option of transitioning in a
`30 staged manner. They have had to decide instead between the
`two solutions, and deploy the solution in their entire network
`communications system in one step. This is a barrier for
`deployment of Internet-based solutions 500/502, since an
`existing working network would be replaced by a yet-
`35 untested new network. Also, for organizations with several
`geographically distributed locations a single step conversion
`is very complex. Some organizations may want a redundant
`Internet-based backup between a few locations while main(cid:173)
`taining the frame relay network for the entire organization.
`It would be an advancement in the art to provide new tools
`and techniques for configuring disparate networks (e.g.,
`frame relay/point-to-point WANs and Internet-based VPNs)
`in parallel, to obtain benefits such as greater reliability,
`improved security, and/or load-balancing. Such improve-
`45 ments are disclosed and claimed herein.
`
`3
`network becomes unavailable. In that case, it may take even
`skilled network administrators several hours to perform the
`steps needed to switch the traffic away from the failed
`network and onto the backup network, unless the invention
`of the '197 application is used. In general, the necessary
`Private Virtual Circuits (PVCs) must be established, routers
`at each site 102 must be reconfigured to use the correct serial
`links and PVCs, and LANs at each site 102 must be
`reconfigured to point at the correct router as the default
`gateway.
`Although two private networks are shown in FIG. 1, three
`or more such networks could be employed, with similar
`considerations coming into play as to increased reliability,
`limits on load-balancing, the efforts needed to switch traffic
`when a network fails, and so on. Likewise, for clarity of 15
`illustration FIG. 1 shows only two sites, but three or more
`sites could communicate through one or more private net(cid:173)
`works.
`FIG. 2 illustrates a prior art configuration in which data is
`normally sent between sites 102 over a private network 106.
`A failover box 202 at each site 102 can detect failure of the
`network 106 and, in response to such a failure, will send the
`data instead over an ISDN link 204 while the network 106
`is down. Using an ISDN link 204 as a backup is relatively
`easier and less expensive than using another private network 25
`106 as the backup, but generally provides lower throughput.
`The ISDN link is an example of a point-to-point or leased
`line network link.
`FIG. 3 illustrates prior art configurations involving two
`private networks for increased reliability, in the sense that
`some of the sites in a given government agency or other
`entity 302 can continue communicating even after one
`network goes down. For instance, if a frame relay network
`A goes down, sites 1, 2, and 3 will be unable to communicate
`with each other but sites 4, 5, and 6 will still be able to
`communicate amongst themselves through frame relay net(cid:173)
`work B. Likewise, if network B goes down, sites 1, 2, and
`3 will still be able to communicate through network A. Only
`if both networks go down at the same time would all sites be
`completely cut off. Like the FIG. 1 configurations, the FIG. 40
`3 configuration uses two private networks. Unlike FIG. 1,
`however, there is no option for switching traffic to another
`private network when one network 106 goes down, although
`either or both of the networks in FIG. 3 could have an ISDN
`backup like that shown in FIG. 2. Note also that even when
`both private networks are up, sites 1, 2, and 3 communicate
`only among themselves; they are not connected to sites 4, 5,
`and 6. Networks A and B in FIG. 3 are therefore not in
`"parallel" as that term is used herein, because all the traffic
`between each pair of sites goes through at most one of the 50
`networks A, B.
`FIG. 4 illustrates a prior art response to the incompatibil-
`ity of frame relay networks of different carriers. A special
`"network-to-network interface" (NNI) 402 is used to reli(cid:173)
`ably transmit data between the two frame relay networks A
`and B. NNis are generally implemented in software at
`carrier offices. Note that the configuration in FIG. 4 does not
`provide additional reliability by using two frame relay
`networks 106, because those networks are in series rather
`than in parallel. If either of the frame relay networks A, B in
`the FIG. 4 configuration fails, there is no path between site
`1 and site 2; adding the second frame relay network has not
`increased reliability. By contrast, FIG. 1 increases reliability
`by placing the frame relay networks in parallel, so that an
`alternate path is available if either (but not both) of the frame 65
`relay networks fails. Someone of skill in the art who was
`looking for ways to improve reliability by putting networks
`
`BRIEF SUMMARY OF THE INVENTION
`
`The present invention provides tools and techniques for
`directing packets over multiple parallel disparate networks,
`based on addresses and other criteria. This helps organiza(cid:173)
`tions make better use of frame relay networks and/or point-
`to-point (e.g., Tl, T3, fiber, OCx, Gigabit, wireless, or
`satellite based) network connections in parallel with VPNs
`and/or other Internet-based networks. For instance, some
`55 embodiments of the invention allow frame relay and VPN
`wide area networks to co-exist for redundancy as well as for
`transitioning from frame relay/point-to-point solutions to
`Internet-based solutions in a staged manner. Some embodi(cid:173)
`ments operate in configurations which communicate data
`60 packets over two or more disparate WAN connections, with
`the data traffic being dynamically load-balanced across the
`connections, while some embodiments treat one of the
`WANs as a backup for use mainly in case the primary
`connection through the other WAN fails.
`Other features and advantages of the invention will
`become more fully apparent through the following descrip(cid:173)
`tion.
`
`Cisco Systems, Inc.
`Exhibit 1002
`Page 17 of 426
`
`

`

`US 6,775,235 B2
`
`5
`BRIEF DESCRIPTION OF THE DRAWINGS
`To illustrate the manner in which the advantages and
`features of the invention are obtained, a more particular
`description of the invention will be given with reference to
`the attached drawings. These drawings only illustrate
`selected aspects of the invention and its context. In the
`drawings:
`FIG. 1 is a diagram illustrating a prior art approach having
`frame relay networks configured in parallel for increased
`reliability for all networked sites, in configurations that
`employ manual switchover between the two frame relay
`networks in case of failure.
`FIG. 2 is a diagram illustrating a prior art approach having
`a frame relay network configured in parallel with an ISDN
`network link for increased reliability for all networked sites.
`FIG. 3 is a diagram illustrating a prior art approach having
`independent and non-parallel frame relay networks, with
`each network connecting several sites but no routine or
`extensive communication between the networks.
`FIG. 4 is a diagram illustrating a prior art approach having
`frame relay networks configured in series through a
`network-to-network interface, with no consequent increase
`in reliability because the networks are in series rather than
`in parallel.
`FIG. 5 is a diagram illustrating a prior art approach having
`a frame relay network configured in parallel with a VPN or
`other Internet-based network that is disparate to the frame
`relay network, but without the fine-grained packet routing of
`the present invention.
`FIG. 6 is a diagram illustrating one system configuration
`of the present invention, in which the Internet and a private
`network are placed in parallel for increased reliability for all
`networked sites, without requiring manual traffic
`switchover, and with the option in some embodiments of
`load balancing between the networks and/or increasing
`security by transmitting packets of a single logical connec(cid:173)
`tion over disparate networks.
`FIG. 7 is a diagram further illustrating a multiple disparate
`network access controller of the present invention, which
`comprises an interface component for each network to
`which the controller connects, and a path selector in the 40
`controller which uses one or more of the following as
`criteria: destination address, network status (up/down), net(cid:173)
`work load, use of a particular network for previous packets
`in a given logical connection or session.
`FIG. 8 is a flowchart illustrating methods of the present
`invention for sending packets using a controller such as the
`one shown in FIG. 7.
`FIG. 9 is a flowchart illustrating methods of the present
`invention for combining connections to send traffic over
`multiple parallel independent disparate networks for reasons
`such as enhanced reliability, load balancing, and/or security.
`FIG. 10 is a diagram illustrating another system configu(cid:173)
`ration of the present inven