(12) United States Patent
`Datta et al.
`
`I lllll llllllll Ill lllll lllll lllll lllll lllll 111111111111111111111111111111111
`
`US006775235B2
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,775,235 B2
`Aug. 10, 2004
`
`(54) TOOLS AND TECHNIQUES FOR
`DIRECTING PACKETS OVER DISPARATE
`NETWORKS
`
`(75)
`
`Inventors: Sanchaita Datta, Salt Lake City, UT
`(US); Ragula Bhaskar, Salt Lake City,
`UT (US)
`
`(73) Assignee: Ragula Systems, Salt Lake City, UT
`(US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 10/361,837
`
`(22) Filed:
`
`Feb. 7, 2003
`
`(65)
`
`Prior Publication Data
`
`US 2003/0147408 Al Aug. 7, 2003
`
`(63)
`
`(60)
`
`(51)
`(52)
`(58)
`
`(56)
`
`Related U.S. Application Data
`
`Continuation-in-part of application No. 10/034,197, filed on
`Dec. 28, 2001.
`Provisional application No. 60/355,509, filed on Feb. 8,
`2002, and provisional application No. 60/259,269, filed on
`Dec. 29, 2000.
`
`Int. Cl.7 ................................................ H04L 12/64
`U.S. Cl. ........................ 370/238; 370/252; 370/352
`Field of Search ................................. 370/252, 352,
`370/230, 235, 238
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,398,012 A
`5,420,862 A
`5,473,599 A
`5,737,526 A
`5,898,673 A
`
`3/1995 Derby et al. ........... 340/825.03
`5/1995 Perlman .................. 370/85.13
`....................... 370/16
`12/1995 Li et al.
`4/1998 Periasamy et al. . ... . 395/200.06
`4/1999 Riggan et al.
`.............. 370/237
`
`................. 709/240
`9/1999 Kitai et al.
`5,948,069 A
`6,016,307 A * 1/2000 Kaplan et al.
`.............. 370/238
`6,119,170 A * 9/2000 Schoffelman et al. ....... 709/244
`6,128,298 A * 10/2000 Wootton et al.
`............ 370/392
`6,253,247 Bl
`6/2001 Bhaskar et al. ............. 709/237
`6,295,276 Bl
`9/2001 Datta et al. ................. 370/218
`6,339,595 Bl
`1/2002 Rekhter et al.
`............. 370/392
`6,438,100 Bl
`8/2002 Halpern et al. ............. 370/218
`6,449,259 Bl
`9/2002 Allain et al. ................ 370/253
`6,456,594 Bl
`9/2002 Kaplan et al.
`.............. 370/238
`6,493,341 Bl
`12/2002 Datta et al. ................. 370/392
`6,493,349 Bl
`12/2002 Casey ........................ 370/409
`6,665,702 Bl * 12/2003 Zisapel et al. .............. 718/105
`
`OTHER PUBLICATIONS
`
`'Radware announces LinkProof: The first IP Load Balancing
`Solution for networks with multiple ISP connection', Press
`Release, published Oct. 7, 1999. *
`'Radware Balances the Network', Internet Traffic Manage(cid:173)
`ment Center, published Jan. 1, 2000. *
`'Global Product Spotlight: Radware Linkproof', Network(cid:173)
`Magazine.com, published Dec. 1, 1999. *
`(List continued on next page.)
`
`Primary Examiner-Melvin Marcelo
`(74) Attorney, Agent, or Firm-Thorpe North & Western
`LLP
`
`(57)
`
`ABSTRACT
`
`Methods, configured storage media, and systems are pro(cid:173)
`vided for communications using two or more disparate
`networks in parallel to provide load balancing across net(cid:173)
`work connections, greater reliability, and/or increased secu(cid:173)
`rity. A controller provides access to two or more disparate
`networks in parallel, through direct or indirect network
`interfaces. When one attached network fails, the failure is
`sensed by the controller and traffic is routed through one or
`more other disparate networks. When all attached disparate
`networks are operating, one controller preferably balances
`the load between them.
`
`24 Claims, 6 Drawing Sheets
`
`INTERNET 500
`
`LINE 4
`
`/LINE 5
`
`ROUTER
`105
`
`ROUTER
`105
`
`FRAME RELAY I POINT-TO-POINT NETWORK 106/204
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 1 of 18
`
`

`

`US 6,775,235 B2
`Page 2
`
`OIBER PUBLICATIONS
`
`'Radware Seeks Solutions to Easy-Access Problems', South
`China Morning Post, published Dec. 7, 1999.*
`B. Gleeson et al., "A Framework for IP Based Virtual Private
`Networks," RFC 2764 (Feb. 2000).
`U.S. patent application, Attorney Docket No. 3003.2.9A; see
`USPTO published application No. US-2002--0087724--Al,
`Jul. 4, 2002.
`T. Liao et al., "Using multiple links to interconnect LANs
`and public circuit switched data networks," Proc. Int. Con(cid:173)
`ference on Communications Systems: Towards Global Inte(cid:173)
`gration, vol. 1, Singapore, 59 Nov. 1990, pp. 289-293.
`Press release from www.coyotepoint,com, Sep. 8, 1997.
`Network Address Translation Technical Discussion, from
`safety.net; no later than May 7, 1999.
`Higginson et al., "Development of Router Clusters to Pro(cid:173)
`vide Fast Failover in IP Networks," from www.asia-paci(cid:173)
`fic.digital.com; no later than Sep. 29, 1998.
`
`Pages from www.navpoint.com; no later than Dec. 24, 2001.
`"The Basic Guide to Frame Relay Networking", pp. 1-85,
`copyright date 1998.
`"NNI & UNI", pp. 1-2, Nov. 16, 2001.
`"Disaster Recovery for Frame Relay Networks", pp. 1-14,
`no later than Dec. 7, 2001.
`T. Nolle, "Watching Your Back", pp. 1-3, Nov. 1, 1999.
`"Multi-Attached and Multi-Homed Dedicated Access", pp.
`1-5, no later than Dec. 8, 2001.
`Feibel, "Internetwork Link," Novell's® Complete Encyclo(cid:173)
`pedia of Networking, copyright date 1995.
`Tanenbaum, Computer Networks (3rd Ed.), pp. 396-406;
`copyright date 1996.
`Wexler, "Frame Relay and IPVPNs: Compete Or Coexist?",
`from www.bcr.com; Jul. 1999.
`
`* cited by examiner
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 2 of 18
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 1 of 6
`
`US 6,775,235 B2
`
`ROUTER A1
`105
`I
`FRAME RELAY
`NETWORK A
`106
`
`I
`ROUTERA2
`105
`
`SITE 1
`102
`
`ROUTER 81
`105
`
`r
`FRAME RELAY
`NETWORK B
`108
`
`1
`
`ROUTER 82
`105
`
`SITE 2
`102
`
`(PRIOR ART)
`Fig. 1
`
`SITE 1
`102
`
`ROUTER 1 105
`
`FAILOVER
`COMPONENT 202
`l
`ISDN NETWORK
`LINK 204
`
`I
`FRAME RELAY
`NETWORK 106
`1
`I
`ROUTER 2 105
`
`FAILOVER
`COMPONENT 202
`
`(PRIOR ART)
`Fig. 2
`
`SITE2
`102
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 3 of 18
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 2 of 6
`
`US 6,775,235 B2
`
`CORPORATION OR OTHER ENTITY 302
`JSITE4l
`fSITE1l
`L__;JJg_J
`L__;JJg_J
`I
`I
`FRAME RELAY
`FRAME RELAY
`NETWORK B
`NETWORK A
`106
`106
`
`~~
`r=b~
`~ ugu
`~~
`(PRIOR ART)
`Fig. 3
`
`SITE 1
`102
`
`ROUTER 1
`105
`I
`FRAME RELAY
`NETWORK A 106
`I
`NETWORK-TO-NETWORK
`INTERFACE 402
`I
`FRAME RELAY
`NETWORK B 106
`I
`ROUTER 2
`105
`
`-----fSiTE2l
`SIT
`L__;JJg_J
`
`(PRIOR ART)
`Fig. 4
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 4 of 18
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 3 of 6
`
`US 6,775,235 B2
`
`ROUTERA1
`105
`
`SITE 1
`102
`
`I
`FRAME RELAY
`NETWORK A
`106
`I
`ROUTERA2
`105
`
`SITE2
`102
`
`ROUTER B1
`104
`I
`INTERNET I VIRTUAL
`PRIVATE NETWORK
`500 / 502
`I
`ROUTER 82
`104
`
`(PRIOR ART)
`Fig. 5
`
`SITE
`102
`
`~
`
`MULTIPLE DISPARATE NETWORK ACCESS
`CONTROLLER 602
`
`SITE INTERFACE 702
`PACKET PATH SELECTOR (E.G., LOAD
`BALANCING, REDUNDANCY, SECURITY) 704
`
`INTERFACE
`706
`
`INTERFACE
`706
`
`INTERFACE
`706
`
`TOA
`NETWORK
`BY PATH
`A1
`
`, ..
`
`,,
`
`TOA
`NETWORK
`BY PATH
`A2
`
`~r
`
`TO
`A
`
`NETW ORK
`TH BYPA
`
`A3
`
`Fig. 7
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 5 of 18
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 4 of 6
`
`US 6,775,235 B2
`
`Cl)
`z
`0
`I-
`()
`
`w z z
`
`I
`
`INTERNET 500
`
`I
`
`/LINE 1 V- LINE 2
`
`V- LINE 3 r- LINE 4
`
`ROUTER ROUTER
`104
`104
`
`ROUTER ROUTER
`104
`104
`
`0
`()
`I-w
`z
`0::::
`w
`I-
`z -
`SITEA CONTROLLER ~ CONTROLLER
`
`VPN
`604
`
`I
`VPN
`604
`
`I
`
`I
`
`602
`
`VPN
`604
`
`I
`
`,_
`
`602
`
`SITE C
`102
`
`/LINE 5
`
`LINE 6 \
`
`LINE 7 \
`
`ROUTER
`105
`
`ROUTER
`105
`
`ROUTER
`105
`
`I
`I
`FRAME RELAY I POINT-TO-POINT NETWORK 106/204
`
`102
`
`-
`
`0::::
`0 w Cl)
`>- z z
`<( - 0
`_J _J I-
`w 0 ()
`o::::ww
`w Cl) z
`~ <( z
`<( ~ 0
`0::::
`()
`LL
`
`Fig. 6
`
`INTERNET 500
`
`I
`
`I
`I
`ROUTERZ
`ROUTER X
`104
`104
`I
`I
`CONTROLLER
`SITEA CONTROLLER
`-
`B 602
`A602
`102
`I
`I
`ROUTER Y
`ROUTER W
`105
`105
`I
`I
`FRAME RELAY NETWORK 106
`Fig. 10
`
`I
`
`-
`
`SITE B
`102
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 6 of 18
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 5 of 6
`
`US 6,775,235 B2
`
`- 1 ..
`
`SPECIFY PATH SELECTOR CRITERIA 800
`......
`SEND PACKET(S) TO CONTROLLER 802
`i
`DETECT NETWORK FAILURE 804
`~
`ROUTE AROUND FAILURE 806
`
`I
`
`I
`
`l+-
`l
`
`I
`
`l
`
`~1
`
`I
`
`ilr
`
`I
`Fig. 8
`
`i
`-.. OBTAIN SYSTEM
`TOPOLOGY
`INFORMATION 902
`
`1
`
`i
`OBTAIN ADDRESS
`RANGE
`INFORMATION 900
`i
`
`-
`....
`
`-1
`
`.....
`RECEIVE PACKET FROM LOCAL SITE 904
`I ..
`LOOK FOR ADDRESS TO "KNOWN" DESTINATION 906
`i
`SELECT PATH TO A DISPARATE NETWORK 908
`USE LOAD BALANCING CRITERION 910
`
`I
`
`USE CONNECTIVITY CRITERION 912
`
`USE SECURITY CRITERION 914
`
`I
`
`I
`
`i
`I MODIFY PACKET DESTINATION ADDRESS 916 I
`i
`FORWARD PACKET ON SELECTED PATH 918
`I
`
`I
`
`Fig. 9
`
`I
`
`I
`
`I
`
`I.__
`ilr
`
`I
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 7 of 18
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 6 of 6
`
`US 6,775,235 B2
`
`I
`
`-
`
`VPNA
`101
`I
`SITEA
`102
`
`I
`
`INTERNET 500
`
`I
`I
`ROUTERZ
`ROUTERX
`104
`104
`I
`I
`CONTROLLER CONTROLLER
`B 602
`A602
`I
`I
`ROUTER W
`ROUTER Y
`105
`105
`I
`I
`FRAME RELAY NETWORK 106
`Fig. 11
`
`I
`
`-
`
`VPN B
`101
`
`I
`SITE B
`102
`
`I
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 8 of 18
`
`

`

`US 6,775,235 B2
`
`1
`TOOLS AND TECHNIQUES FOR
`DIRECTING PACKETS OVER DISPARATE
`NETWORKS
`
`RELATED APPLICATIONS
`
`This application claims priority to commonly owned
`copending U.S. provisional patent application serial No.
`60/355,509 filed Feb. 8, 2002, which is also incorporated
`herein by reference. This application is a continuation-in(cid:173)
`part of U.S. patent application Ser. No. 10/034,197 filed
`Dec. 28, 2001, which claims priority to U.S. provisional
`patent application serial No. 60/259,269 filed Dec. 29, 2000,
`each of which is also incorporated herein by reference.
`
`FIELD OF THE INVENTION
`
`The present invention relates to computer network data
`transmission, and more particularly relates to tools and
`techniques for communications using disparate parallel
`networks, such as a virtual private network ("VPN") or the
`Internet in parallel with a point-to-point, leased line, or
`frame relay network, in order to help provide benefits such
`as load balancing across network connections, greater
`reliability, and increased security.
`
`TECHNICAL BACKGROUND OF THE
`INVENTION
`
`Organizations have used frame relay networks and point(cid:173)
`to-point leased line networks for interconnecting geographi(cid:173)
`cally dispersed offices or locations. These networks have
`been implemented in the past and are currently in use for
`interoffice communication, data exchange and file sharing.
`Such networks have advantages, some of which are noted
`below. But these networks also tend to be expensive, and 35
`there are relatively few options for reliability and redun(cid:173)
`dancy. As networked data communication becomes critical
`to the day-to-day operation and functioning of an
`organization, the need for lower cost alternatives for redun(cid:173)
`dant back-up for wide area networks becomes important.
`Frame relay networking technology offers relatively high
`throughput and reliability. Data is sent in variable length
`frames, which are a type of packet. Each frame has an
`address that the frame relay network uses to determine the
`frame's destination. The frames travel to their destination 45
`through a series of switches in the frame relay network,
`which is sometimes called a network "cloud"; frame relay is
`an example of packet-switched networking technology. The
`transmission lines in the frame relay cloud must be essen(cid:173)
`tially error-free for frame relay to perform well, although 50
`error handling by other mechanisms at the data source and
`destination can compensate to some extent for lower line
`reliability. Frame relay and/or point-to-point network ser(cid:173)
`vices are provided or have been provided by various carriers,
`such as AT&T, Qwest, XO, and MCI WorldCom.
`Frame relay networks are an example of a network that is
`"disparate" from the Internet and from Internet-based virtual
`private networks for purposes of the present invention.
`Another example of such a "disparate" network is a point(cid:173)
`to-point network, such as a Tl or T3 connection. Although
`the underlying technologies differ somewhat, for purposes
`of the present invention frame relay networks and point-to(cid:173)
`point networks are generally equivalent in important ways,
`such as the conventional reliance on manual switchovers
`when traffic must be redirected after a connection fails, and
`their implementation distinct from the Internet. A frame
`relay permanent virtual circuit is a virtual point-to-point
`
`2
`connection. Frame relays are used as examples throughout
`this document, but the teachings will also be understood in
`the context of point-to-point networks.
`A frame relay or point-to-point network may become
`5 suddenly unavailable for use. For instance, both MCI World(cid:173)
`Com and AT&T users have lost access to their respective
`frame relay networks during major outages. During each
`outage, the entire network failed. Loss of a particular line or
`node in a network is relatively easy to work around. But loss
`10 of an entire network creates much larger problems.
`Tools and techniques to permit continued data transmis(cid:173)
`sion after loss of an entire frame relay network that would
`normally carry data are discussed in U.S. patent application
`Ser. No. 10/034,197 filed Dec. 28, 2001 and incorporated
`15 herein. The '197 application focuses on architectures involv(cid:173)
`ing two or more "private" networks in parallel, whereas the
`present application focuses on architectures involving dis(cid:173)
`parate networks in parallel, such as a proprietary frame relay
`network and the Internet. Note that the term "private net-
`20 work" is used herein in a manner consistent with its use in
`the '197 application (which comprises frame relay and
`point-to-point networks), except that a "virtual private net(cid:173)
`work" as discussed herein is not a "private network". Virtual
`private networks are Internet-based, and hence disparate
`25 from private networks, i.e., from frame relay and point-to-
`point networks. To reduce the risk of confusion that might
`arise from misunderstanding "private network" to comprise
`"virtual private network" herein, virtual private networks
`will be henceforth referred to as VPNs. Other differences
`30 and similarities between the present application and the '197
`application will also be apparent to those of skill in the art
`on reading the two applications.
`Various architectures involving multiple networks are
`known in the art. For instance, FIG. 1 illustrates prior art
`configurations involving two frame relay networks for
`increased reliability; similar configurations involve one or
`more point-to-point network connections. Two sites 102
`transmit data to each other (alternately, one site might be
`40 only a data source, while the other is only a data destination).
`Each site has two border routers 105. Two frame relay
`networks 106, 108 are available to the sites 102 through the
`routers 105. The two frame relay networks 106, 108 have
`been given separate numbers in the figure, even though each
`is a frame relay network, to emphasize the incompatibility of
`frame relay networks provided by different carriers. An
`AT&T frame relay network, for instance, is incompatible(cid:173)
`in details such as maximum frame size or switching
`capacity-with an MCI WorldCom frame relay network,
`even though they are similar when one takes the broader
`view that encompasses disparate networks like those dis-
`cussed herein. The two frame relay providers have to agree
`upon information rates, switching capacities, frame sizes,
`etc. before the two networks can communicate directly with
`55 each other.
`A configuration like that shown in FIG. 1 may be actively
`and routinely using both frame relay networks A and B. For
`instance, a local area network (LAN) at site 1 may be set up
`to send all traffic from the accounting and sales departments
`60 to router Al and send all traffic from the engineering
`department to router Bl. This may provide a very rough
`balance of the traffic load between the routers, but it does not
`attempt to balance router loads dynamically in response to
`actual traffic and thus is not "load-balancing" as that term is
`65 used herein.
`Alternatively, one of the frame relay networks may be a
`backup which is used only when the other frame relay
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 9 of 18
`
`

`

`US 6,775,235 B2
`
`3
`4
`network becomes unavailable. In that case, it may take even
`in parallel would probably not consider NNis pertinent,
`skilled network administrators several hours to perform the
`because they were used for serial configurations rather than
`steps needed to switch the traffic away from the failed
`parallel ones, and adding networks in a serial manner does
`network and onto the backup network, unless the invention
`not improve reliability.
`of the '197 application is used. In general, the necessary 5
`Internet-based communication solutions such as VPNs
`Private Virtual Circuits (PVCs) must be established, routers
`and Secure Sockets Layer (SSL) offer alternatives to frame
`at each site 102 must be reconfigured to use the correct serial
`relay 106 and point-to-point leased line networks such as
`links and PVCs, and LANs at each site 102 must be
`those using an ISDN link 204. These Internet-based solu(cid:173)
`reconfigured to point at the correct router as the default
`tions are advantageous in the flexibility and choice they offer
`gateway.
`Although two private networks are shown in FIG. 1, three 10 in cost, in service providers, and in vendors. Accordingly,
`some organizations have a frame relay 106 or leased line
`or more such networks could be employed, with similar
`connection (a.k.a. point-to-point) for intranet communica(cid:173)
`considerations coming into play as to increased reliability,
`tion and also have a connection for accessing the Internet
`limits on load-balancing, the efforts needed to switch traffic
`500, using an architecture such as that shown in FIG. 5.
`when a network fails, and so on. Likewise, for clarity of 15
`But better tools and techniques are needed for use in
`illustration FIG. 1 shows only two sites, but three or more
`architectures such as that shown in FIG. 5. In particular,
`sites could communicate through one or more private net(cid:173)
`prior approaches for selecting which network to use for
`works.
`which packet(s) are coarse. For instance, all packets from
`FIG. 2 illustrates a prior art configuration in which data is
`department X might be sent over the frame relay connection
`normally sent between sites 102 over a private network 106. 20
`106 while all packets from department Y are sent over the
`A failover box 202 at each site 102 can detect failure of the
`Internet 500. Or the architecture might send all traffic over
`network 106 and, in response to such a failure, will send the
`the frame relay network unless that network fails, and then
`data instead over an ISDN link 204 while the network 106
`be manually reconfigured to send all traffic over a VPN 502.
`is down. Using an ISDN link 204 as a backup is relatively
`Organizations are still looking for better ways to use
`easier and less expensive than using another private network 25
`Internet-based redundant connections to backup the primary
`106 as the backup, but generally provides lower throughput.
`frame relay networks. Also, organizations wanting to change
`The ISDN link is an example of a point-to-point or leased
`from frame relay and point-to-point solutions to Internet(cid:173)
`line network link.
`based solutions have not had the option of transitioning in a
`FIG. 3 illustrates prior art configurations involving two
`30 staged manner. They have had to decide instead between the
`private networks for increased reliability, in the sense that
`two solutions, and deploy the solution in their entire network
`some of the sites in a given government agency or other
`communications system in one step. This is a barrier for
`entity 302 can continue communicating even after one
`deployment of Internet-based solutions 500/502, since an
`network goes down. For instance, if a frame relay network
`existing working network would be replaced by a yet-
`A goes down, sites 1, 2, and 3 will be unable to communicate
`35 untested new network. Also, for organizations with several
`with each other but sites 4, 5, and 6 will still be able to
`geographically distributed locations a single step conversion
`communicate amongst themselves through frame relay net(cid:173)
`is very complex. Some organizations may want a redundant
`work B. Likewise, if network B goes down, sites 1, 2, and
`Internet-based backup between a few locations while main(cid:173)
`3 will still be able to communicate through network A. Only
`taining the frame relay network for the entire organization.
`if both networks go down at the same time would all sites be
`It would be an advancement in the art to provide new tools
`completely cut off. Like the FIG. 1 configurations, the FIG. 40
`and techniques for configuring disparate networks (e.g.,
`3 configuration uses two private networks. Unlike FIG. 1,
`frame relay/point-to-point WANs and Internet-based VPNs)
`however, there is no option for switching traffic to another
`in parallel, to obtain benefits such as greater reliability,
`private network when one network 106 goes down, although
`improved security, and/or load-balancing. Such improve-
`either or both of the networks in FIG. 3 could have an ISDN
`45 ments are disclosed and claimed herein.
`backup like that shown in FIG. 2. Note also that even when
`both private networks are up, sites 1, 2, and 3 communicate
`only among themselves; they are not connected to sites 4, 5,
`and 6. Networks A and B in FIG. 3 are therefore not in
`"parallel" as that term is used herein, because all the traffic
`between each pair of sites goes through at most one of the 50
`networks A, B.
`FIG. 4 illustrates a prior art response to the incompatibil-
`ity of frame relay networks of different carriers. A special
`"network-to-network interface" (NNI) 402 is used to reli(cid:173)
`ably transmit data between the two frame relay networks A
`and B. NNis are generally implemented in software at
`carrier offices. Note that the configuration in FIG. 4 does not
`provide additional reliability by using two frame relay
`networks 106, because those networks are in series rather
`than in parallel. If either of the frame relay networks A, B in
`the FIG. 4 configuration fails, there is no path between site
`1 and site 2; adding the second frame relay network has not
`increased reliability. By contrast, FIG. 1 increases reliability
`by placing the frame relay networks in parallel, so that an
`alternate path is available if either (but not both) of the frame 65
`relay networks fails. Someone of skill in the art who was
`looking for ways to improve reliability by putting networks
`
`The present invention provides tools and techniques for
`directing packets over multiple parallel disparate networks,
`based on addresses and other criteria. This helps organiza(cid:173)
`tions make better use of frame relay networks and/or point-
`to-point (e.g., Tl, T3, fiber, OCx, Gigabit, wireless, or
`satellite based) network connections in parallel with VPNs
`and/or other Internet-based networks. For instance, some
`55 embodiments of the invention allow frame relay and VPN
`wide area networks to co-exist for redundancy as well as for
`transitioning from frame relay/point-to-point solutions to
`Internet-based solutions in a staged manner. Some embodi(cid:173)
`ments operate in configurations which communicate data
`60 packets over two or more disparate WAN connections, with
`the data traffic being dynamically load-balanced across the
`connections, while some embodiments treat one of the
`WANs as a backup for use mainly in case the primary
`connection through the other WAN fails.
`Other features and advantages of the invention will
`become more fully apparent through the following descrip(cid:173)
`tion.
`
`BRIEF SUMMARY OF THE INVENTION
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 10 of 18
`
`

`

`US 6,775,235 B2
`
`5
`
`5
`BRIEF DESCRIPTION OF THE DRAWINGS
`To illustrate the manner in which the advantages and
`features of the invention are obtained, a more particular
`description of the invention will be given with reference to
`the attached drawings. These drawings only illustrate
`selected aspects of the invention and its context. In the
`drawings:
`FIG. 1 is a diagram illustrating a prior art approach having
`frame relay networks configured in parallel for increased
`reliability for all networked sites, in configurations that
`employ manual switchover between the two frame relay
`networks in case of failure.
`FIG. 2 is a diagram illustrating a prior art approach having
`a frame relay network configured in parallel with an ISDN
`network link for increased reliability for all networked sites.
`FIG. 3 is a diagram illustrating a prior art approach having
`independent and non-parallel frame relay networks, with
`each network connecting several sites but no routine or
`extensive communication between the networks.
`FIG. 4 is a diagram illustrating a prior art approach having
`frame relay networks configured in series through a
`network-to-network interface, with no consequent increase
`in reliability because the networks are in series rather than
`in parallel.
`FIG. 5 is a diagram illustrating a prior art approach having
`a frame relay network configured in parallel with a VPN or
`other Internet-based network that is disparate to the frame
`relay network, but without the fine-grained packet routing of
`the present invention.
`FIG. 6 is a diagram illustrating one system configuration
`of the present invention, in which the Internet and a private
`network are placed in parallel for increased reliability for all
`networked sites, without requiring manual traffic
`switchover, and with the option in some embodiments of
`load balancing between the networks and/or increasing
`security by transmitting packets of a single logical connec(cid:173)
`tion over disparate networks.
`FIG. 7 is a diagram further illustrating a multiple disparate
`network access controller of the present invention, which
`comprises an interface component for each network to
`which the controller connects, and a path selector in the 40
`controller which uses one or more of the following as
`criteria: destination address, network status (up/down), net(cid:173)
`work load, use of a particular network for previous packets
`in a given logical connection or session.
`FIG. 8 is a flowchart illustrating methods of the present
`invention for sending packets using a controller such as the
`one shown in FIG. 7.
`FIG. 9 is a flowchart illustrating methods of the present
`invention for combining connections to send traffic over
`multiple parallel independent disparate networks for reasons
`such as enhanced reliability, load balancing, and/or security.
`FIG. 10 is a diagram illustrating another system configu(cid:173)
`ration of the present invention, in which the Internet and a
`frame relay network are placed in parallel, with a VPN
`tunnel originating after the source controller and terminating 55
`before the destination controller, and each known site that is
`accessible through one network is also accessible through
`the other network unless that other network fails.
`FIG. 11 is a diagram illustrating a system configuration
`similar to FIG. 10, except the VPN tunnel originates before
`the source controller and terminates after the destination
`controller.
`
`6
`independent parallel disparate networks, such as frame relay
`networks and/or point-to-point network connections, on the
`one hand, and VPNs or other Internet-based network
`connections, on the other hand. "Multiple" networks means
`two or more such networks. "Independent" means routing
`information need not be shared between the networks.
`"Parallel" does not rule out all use of NNis and serial
`networks, but it does require that at least two of the networks
`in the configuration be in parallel at the location where the
`10 invention distributes traffic, so that alternate data paths
`through different networks are present. "Frame relay net(cid:173)
`works" or "private networks" does not rule out the use of an
`ISDN link or other backup for a particular frame relay or
`point-to-point private network, but it does require the pres-
`15 ence of multiple such networks; FIG. 2, for instance, does
`not meet this requirement. A "frame relay network" is
`unavailable to the general public and thus disparate from the
`Internet and VPNs (which may be Internet-based), even
`though some traffic in the Internet may use public frame
`20 relay networks once the traffic leaves the location where the
`invention distributes traffic.
`FIG. 6 illustrates one of many possible configurations of
`the present invention. Comments made here also apply to
`similar configurations involving only one or more frame
`25 relay networks 106, those involving only one or more
`point-to-point networks 204, and those not involving a VPN
`604, for example. Two or more disparate networks are
`placed in parallel between two or more sites 102. In the
`illustrated configuration, the Internet 500 and a VPN 604 are
`30 disparate from, and in parallel with, frame relay/point-to(cid:173)
`point network 106/204, with respect to site A and site B. No
`networks are parallel disparate networks in FIG. 6 with
`regard to site C as a traffic source, since that site is not
`connected to the Internet 500. Access to the disparate
`35 networks at site A and and site B is through an inventive
`controller 602 at each site. Additional controllers 602 may
`be used at each location (i.e., controllers 602 may be placed
`in parallel to one another) in order to provide a switched
`connection system with no single point of failure.
`With continued attention to the illustrative network topol-
`ogy for one embodiment of the invention shown in FIG. 6,
`in this topology the three locations A, B, and C are con(cid:173)
`nected to each other via a frame relay 106 or leased line
`network 204. Assume, for example, that all three locations
`45 are connected via a single frame relay network 106. Loca(cid:173)
`tions A and B are also connected to each other via a VPN
`connection 604. VPN tunnels are established between loca(cid:173)
`tions A and B in the VPN, which pairs line 1 to line 3 and
`also pairs line 2 to line 3. There can be only one VPN tunnel
`50 between locations A and B. There is no VPN connection
`between location C and either location A or location B.
`Therefore, locations A, B, and C can communicate with
`each other over the frame relay network 106, and locations
`A and B (but not C) can also communicate with each other
`over the VPN connection 604. Communication between
`locations A and C, and communication between locations B
`and C, can take place over the frame relay network 106 only.
`Communication between locations A and B can take place
`over frame relay network 106. It can also take place over one
`60 of the lines 1-and-3 pair, or the lines 2-and-3 pair, but not
`both at the same time. Traffic can also travel over lines 2 and
`4, but without a VPN tunnel. When the source and destina(cid:173)
`tion IP address pairs are the same between locations A and
`B but different types of networks connect those locations, as
`65 in FIG. 6 for instance, then a traffic routing decision that
`selects between network types cannot be made with an
`existing commercially available device. By contrast, the
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`The present invention relates to methods, systems, and
`configured storage media for connecting sites over multiple
`
`Cisco Systems, Inc.
`Exhibit 1001
`Page 11 of 18
`
`

`

`US 6,775,235 B2
`
`5
`
`7
`invention allows an organization to deploy an Internet-based
`solution between locations A and B while maintaining the
`frame relay network 106 between locations A, B, and C, and
`allows traffic routing that selects between the Internet and
`the frame relay network on a packet-by-packet basis.
`The invention may thus be configured to allow the orga(cid:173)
`nization to achieve the following goals, in the context of
`FIG. 6; similar goals are facilitated in other configurations.
`First, the organization can deploy an Internet-based second
`connection between only locations A and B, while main- 10
`taining frame relay connectivity between locations A, B, and
`C. Later the organization may deploy an Internet-based
`solution at location C as well. Second, the organization can
`use the Internet-based connection between locations A and B
`for full l