I lllll llllllll Ill lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US007 406048B2
`
`c12) United States Patent
`Datta et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 7 ,406,048 B2
`Jul. 29, 2008
`
`(54) TOOLS AND TECHNIQUES FOR DIRECTING
`PACKETS OVER DISPARATE NETWORKS
`
`(76)
`
`Inventors: Sanchaita Datta, 4540 S. Jupiter Dr.,
`Salt Lake City, UT (US) 84124; Bhaskar
`Ragula, 4540 S. Jupiter Dr., Salt Lake
`City, UT (US) 84124
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 821 days.
`
`6,016,307 A *
`1/2000 Kaplan et al ................ 370/238
`6/2001 Bhaskar et al.
`6,253,247 Bl
`9/2001 Datta et al.
`6,295,276 Bl
`6,339,595 Bl
`1/2002 Rekhter et al.
`8/2002 Halpern et al.
`6,438,100 Bl
`912002 Allain et al.
`6,449,259 Bl
`912002 Kaplan et al.
`6,456,594 Bl
`12/2002 Datta et al.
`6,493,341 Bl
`12/2002 Casey
`6,493,349 Bl
`7,088,716 B2 * 8/2006 Sugai et al. ................. 370/392
`2002/0087724 Al
`712002 Datta et al.
`
`(21) Appl. No.: 10/911,846
`
`(22) Filed:
`
`Aug. 3, 2004
`
`(65)
`
`Prior Publication Data
`
`US 2005/0008017 Al
`
`Jan. 13, 2005
`
`Related U.S. Application Data
`
`(63)
`
`(60)
`
`Continuation-in-part of application No. 10/034,197,
`filed on Dec. 28, 2001.
`
`Provisional application No. 601259,269, filed on Dec.
`29, 2000.
`
`(51)
`
`Int. Cl.
`(2006.01)
`H04L 12164
`(52) U.S. Cl. ........................ 370/238; 370/252; 370/352
`( 58) Field of Classification Search .... ...... ....... 3 70/238,
`370/252, 352
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,398,012 A
`5,420,862 A
`5,473,599 A
`5,737,526 A
`5,898,673 A
`5,948,069 A
`
`3/1995 Derby et al.
`5/1995 Perlman
`12/1995 Li et al.
`4/1998 Periasamy et al.
`4/1999 Riggan et al.
`9/1999 Kitai et al.
`
`OTHER PUBLICATIONS
`
`T. Liao et al., "Using multiple links to interconnect LAN sand public
`circuit switched data networks," Proc. Int. Conference on Commu(cid:173)
`nications Systems: Towards Global Integration, vol. 1, Singapore, 59
`Nov. 1990, pp. 289-293.
`
`(Continued)
`
`Primary Examiner-Melvin Marcelo
`(74) Attorney, Agent, or Firm-Ogilvie Law Firm
`
`(57)
`
`ABSTRACT
`
`Methods, configured storage media, and systems are pro(cid:173)
`vided for communications using two or more disparate net(cid:173)
`works in parallel to provide load balancing across network
`connections, greater reliability, and/or increased security. A
`controller provides access to two or more disparate networks
`in parallel, through direct or indirect network interfaces.
`When one attached network fails, the failure is sensed by the
`controller and traffic is routed through one or more other
`disparate networks. When all attached disparate networks are
`operating, one controller preferably balances the load
`between them.
`
`24 Claims, 6 Drawing Sheets
`
`INTERNET 500
`
`ROUTER
`105
`
`FRAME RELAY I POINT-TO-POINT NETWORK 106/204
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 1 of 17
`
`

`

`US 7,406,048 B2
`Page 2
`
`OTHER PUBLICATIONS
`Press release from www.coyotepoint.com, Sep. 8, 1997.
`Network Address Translation Technical Discussion, from safety.net;
`no later than May 7, 1999.
`Higginson et al., "Development of Router Clusters to Provide Fast
`Failover in IP Networks," from www.asia-pacific.digital.com; no
`later than Sep. 29, 1998.
`Pages from www.navpoint.com; no later than Dec. 24, 2001.
`"The Basic Guide to Frame Relay Networking", pp. 1-85, copyright
`date 1998.
`"NNI & UNI", pp. 1-2, Nov. 16, 2001.
`"Disaster Recovery for Frame Relay Networks", pp. 1-14, no later
`than Dec. 7, 2001.
`
`T. Nolle, "Watching Your Back", pp. 1-3, Nov. 1, 1999.
`Multi-Attached and Multi-Homed Dedicated Access:, pp. 1-5, no
`later than Dec. 8, 2001.
`Feibel, "Internetwork Link," Novel's® Complete Encyclopedia of
`Networking, copyright date 1995.
`Tanenbaum, Computer Networks (3'a Ed.), pp. 396-406; copyright
`date 1996.
`Wexler, "Frame Relay and IP VPNs: Complete Ore Coexist?", from
`www.bcr.com; Jul. 1999.
`B. Gleeson et al., "A Framework for IP Based Virtual Private Net(cid:173)
`works," RFC 2764 (Feb. 2000).
`* cited by examiner
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 2 of 17
`
`

`

`U.S. Patent
`
`Jul. 29, 2008
`
`Sheet 1of6
`
`US 7 ,406,048 B2
`
`ROUTER A1
`105
`I
`FRAME RELAY
`NETWORK A
`106
`I
`ROUTERA2
`105
`
`SITE 1
`102
`
`SITE2
`102
`
`ROUTER 81
`105
`I
`FRAME RELAY
`NETWORK 8
`108
`I
`ROUTER 82
`105
`
`(PRIOR ART)
`Fig. 1
`
`SITE 1
`102
`
`ROUTER 1 105
`
`FAILOVER
`COMPONENT 202
`I
`ISDN NETWORK
`LINK 204
`
`I
`FRAME RELAY
`NETWORK 106
`I
`I
`ROUTER 2 105
`
`FAILOVER
`COMPONENT 202
`
`(PRIOR ART)
`Fig. 2
`
`SITE2
`102
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 3 of 17
`
`

`

`U.S. Patent
`
`Jul. 29, 2008
`
`Sheet 2 of 6
`
`US 7,406,048 B2
`
`CORPORATION OR OTHER ENTITY 302
`
`I SITE 41
`[SiTE1l
`~ 102
`I
`I
`FRAME RELAY
`FRAME RELAY
`NETWORK A
`NETWORK B
`1Q§
`.1Q§
`
`~
`~
`
`~ ~
`LJi:J
`
`~ ~
`~
`(PRIOR ART)
`Fig. 3
`
`SITE 1
`102
`
`ROUTER 1
`105
`I
`FRAME RELAY
`NETWORK A 106
`I
`NETWORK-TO-NETWORK
`INTERFACE 402
`l
`FRAME RELAY
`NETWORK B 106
`I
`ROUTER 2
`105
`
`-
`
`SITE2
`102
`
`(PRIOR ART)
`Fig. 4
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 4 of 17
`
`

`

`U.S. Patent
`
`Jul. 29, 2008
`
`Sheet 3 of 6
`
`US 7 ,406,048 B2
`
`ROUTER A1
`105
`I
`FRAME RELAY
`NETWORK A
`106
`I
`ROUTERA2
`105
`
`SITE 1
`102
`
`SITE2
`102
`
`ROUTER 81
`104
`I
`INTERNET I VIRTUAL
`PRIVATE NETWORK
`500 / 502
`I
`ROUTER 82
`104
`
`(PRIOR ART)
`Fig. 5
`
`SITE
`102
`
`,___
`
`MULTIPLE DISPARATE NETWORK ACCESS
`CONTROLLER 602
`
`SITE INTERFACE 702
`PACKET PATH SELECTOR (E.G., LOAD
`BALANCING, REDUNDANCY, SECURITY} 704
`
`INTERFACE
`706
`
`INTERFACE
`706
`
`INTERFACE
`706
`
`TOA
`NETWORK
`BY PATH
`A1
`
`,,
`
`,,
`
`TOA
`NETWORK
`BY PATH
`A2
`
`1r
`
`TOA
`NETWO RK
`TH
`BYPA
`A3
`
`Fig. 7
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 5 of 17
`
`

`

`U.S. Patent
`
`Jul. 29, 2008
`
`Sheet 4 of 6
`
`US 7 ,406,048 B2
`
`INTERNET 500
`
`1
`
`/LINE 1 V- LINE 2
`
`V- LINE 3 ,r- LINE 4
`
`ROUTER ROUTER
`104
`104
`
`ROUTER ROUTER
`104
`104
`
`VPN
`604
`
`\
`SITE 8 CONTROLLER
`602
`102 ~
`
`SITEC
`102
`
`VPN
`604
`
`VPN
`604
`
`I
`
`CONTROLLER
`602
`
`(f) I
`z
`0
`I-
`0
`w
`z
`z
`0
`0
`I-w
`z
`a:: w
`I-z -
`
`SITEA
`102
`
`' - -
`
`a::
`0 w (f)
`>- z z
`<( - 0
`_J _J I-
`woo
`a:: w w
`w (/) z
`~ <( z
`<( ~ 0
`a::
`0
`LL
`
`/LINE 5
`
`LINE 6 \
`
`LINE 7 ~
`
`ROUTER
`105
`
`ROUTER
`ROUTER
`105
`105
`I
`I
`FRAME RELAY I POINT-TO-POINT NETWORK 106/204
`
`I
`
`Fig. 6
`
`INTERNET 500
`
`102 -
`
`I
`
`I
`
`I
`I
`ROUTER Z
`ROUTER X
`104
`104
`I
`I
`CONTROLLER
`SITEA CONTROLLER
`B 602
`A602
`I
`I
`ROUTER Y
`ROUTER W
`105
`105
`I
`I
`FRAME RELAY NETWORK 106
`Fig. 10
`
`' - -
`
`SITE 8
`102
`
`l
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 6 of 17
`
`

`

`U.S. Patent
`
`Jul. 29, 2008
`
`Sheet 5 of 6
`
`US 7,406,048 B2
`
`~ SPECIFY PATH SELECTOR CRITERIA 800
`_1_
`~ .... -
`SEND PACKET(S) TO CONTROLLER 802
`
`I
`
`DETECT NETWORK FAILURE 804
`
`I
`
`I
`
`,,
`
`i
`+
`ROUTE AROUND FAILURE 806
`I
`Fig. 8
`
`I+-
`
`I
`
`I
`I
`
`i
`OBTAIN ADDRESS
`RANGE
`INFORMATION 900
`~
`
`~
`
`i
`- OBTAIN SYSTEM
`TOPOLOGY
`INFORMATION 902
`i
`
`-
`-
`
`.... 1
`~.
`
`RECEIVE PACKET FROM LOCAL SITE 904
`
`I
`I •
`LOOK FOR ADDRESS TO "KNOWN" DESTINATION 906 I
`+
`
`SELECT PATH TO A DISPARATE NETWORK 908
`I
`USE LOAD BALANCING CRITERION 910
`I
`I
`
`USE CONNECTIVITY CRITERION 912
`
`USE SECURITY CRITERION 914
`
`i
`I MODIFY PACKET DESTINATION ADDRESS 916 I
`i
`FORWARD PACKET ON SELECTED PATH 918
`I +
`Fig. 9
`
`I
`
`I
`I
`I
`
`+-,,
`I
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 7 of 17
`
`

`

`U.S. Patent
`
`Jul. 29, 2008
`
`Sheet 6 of 6
`
`US 7,406,048 B2
`
`I
`
`,___
`
`VPNA
`101
`I
`SITEA
`102
`
`I
`
`INTERNET 500
`
`I
`
`I
`I
`ROUTER Z
`ROUTERX
`104
`104
`I
`I
`CONTROLLER CONTROLLER
`VPN B
`B 602 ~ 101
`A602
`I
`I
`ROUTER Y
`ROUTERW
`105
`105
`I
`I
`FRAME RELAY NETWORK 106
`Fig. 11
`
`I
`SITE B
`102
`
`I
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 8 of 17
`
`

`

`US 7,406,048 B2
`
`1
`TOOLS AND TECHNIQUES FOR DIRECTING
`PACKETS OVER DISPARATE NETWORKS
`
`RELATED APPLICATIONS
`
`This application is a continuation of forthcoming U.S. Pat.
`No. 6,775,235 (application Ser. No. 10/361,837), which
`claims priority to commonly owned U.S. provisional patent
`application Ser. No. 60/355,509 filed Feb. 8,2002, which
`provisional application is also incorporated herein by refer(cid:173)
`ence. U.S. Pat. No. 6,775,235 is a continuation-in-part ofU.S.
`patent application Ser. No. 10/034,197 filed Dec. 28, 2001,
`which claims priority to U.S. provisional patent application
`Ser. No. 601259,269 filed Dec. 29, 2000, each of which appli(cid:173)
`cations is also incorporated herein by reference.
`
`FIELD OF THE INVENTION
`
`The present invention relates to computer network data
`transmission, and more particularly relates to tools and tech(cid:173)
`niques for communications using disparate parallel networks,
`such as a virtual private network ("VPN") or the Internet in
`parallel with a point-to-point, leased line, or frame relay
`network, in order to help provide benefits such as load bal(cid:173)
`ancing across network connections, greater reliability, and
`increased security.
`
`TECHNICAL BACKGROUND OF THE
`INVENTION
`
`Organizations have used frame relay networks and point(cid:173)
`to-point leased line networks for interconnecting geographi(cid:173)
`cally dispersed offices or locations. These networks have
`been implemented in the past and are currently in use for
`interoffice communication, data exchange and file sharing. 35
`Such networks have advantages, some of which are noted
`below. But these networks also tend to be expensive, and there
`are relatively few options for reliability and redundancy. As
`networked data communication becomes critical to the day(cid:173)
`to-day operation and functioning of an organization, the need
`for lower cost alternatives for redundant back-up for wide
`area networks becomes important.
`Frame relay networking technology offers relatively high
`throughput and reliability. Data is sent in variable length
`frames, which are a type of packet. Each frame has an address
`that the frame relay network uses to determine the frame's
`destination. The frames travel to their destination through a
`series of switches in the frame relay network, which is some(cid:173)
`times called a network "cloud"; frame relay is an example of
`packet-switched networking technology. The transmission
`lines in the frame relay cloud must be essentially error-free
`for frame relay to perform well, although error handling by
`other mechanisms at the data source and destination can
`compensate to some extent for lower line reliability. Frame
`relay and/or point-to-point network services are provided or
`have been provided by various carriers, such as AT&T, Qwest,
`XO, and MCI WorldCom.
`Frame relay networks are an example of a network that is
`"disparate" from the Internet and from Internet-based virtual
`private networks for purposes of the present invention.
`Another example of such a "disparate" network is a point-to(cid:173)
`point network, such as a Tl or T3 connection. Although the
`underlying technologies differ somewhat, for purposes of the
`present invention frame relay networks and point-to-point
`networks are generally equivalent in important ways, such as 65
`the conventional reliance on manual switchovers when traffic
`must be redirected after a connection fails, and their imple-
`
`2
`mentation distinct from the Internet. A frame relay permanent
`virtual circuit is a virtual point-to-point connection. Frame
`relays are used as examples throughout this document, but the
`teachings will also be understood in the context of point-to(cid:173)
`point networks.
`A frame relay or point-to-point network may become sud(cid:173)
`denly unavailable for use. For instance, both MCI WorldCom
`and AT&T users have lost access to their respective frame
`relay networks during major outages. During each outage, the
`10 entire network failed. Loss of a particular line or node in a
`network is relatively easy to work around. But loss of an entire
`network creates much larger problems.
`Tools and techniques to permit continued data transmis(cid:173)
`sion after loss of an entire frame relay network that would
`15 normally carry data are discussed in U.S. patent application
`Ser. No. 10/034,197 filed Dec. 28, 2001 and incorporated
`herein. The' 197 application focuses on architectures involv(cid:173)
`ing two or more "private" networks in parallel, whereas the
`present application focuses on architectures involving dispar-
`20 ate networks in parallel, such as a proprietary frame relay
`network and the Internet. Note that the term "private net(cid:173)
`work" is used herein in a manner consistent with its use in the
`'197 application (which comprises frame relay and point-to(cid:173)
`point networks), except that a "virtual private network" as
`25 discussed herein is not a "private network". Virtual private
`networks are Internet-based, and hence disparate from private
`networks, i.e., from frame relay and point-to-point networks.
`To reduce the risk of confusion that might arise from misun(cid:173)
`derstanding "private network" to comprise "virtual private
`30 network" herein, virtual private networks will be henceforth
`referred to as VPNs. Other differences and similarities
`between the present application and the '197 application will
`also be apparent to those of skill in the art on reading the two
`applications.
`Various architectures involving multiple networks are
`known in the art. For instance, FIG. 1 illustrates prior art
`configurations involving two frame relay networks for
`increased reliability; similar configurations involve one or
`more point-to-point network connections. Two sites 102
`40 transmit data to each other (alternately, one site might be only
`a data source, while the other is only a data destination). Each
`site has two border routers 105. Two frame relay networks
`106, 108 are available to the sites 102 through the routers 105.
`The two frame relay networks 106, 108 have been given
`45 separate numbers in the figure, even though each is a frame
`relay network, to emphasize the incompatibility of frame
`relay networks provided by different carriers. An AT&T
`frame relay network, for instance, is incompatible-in details
`such as maximum frame size or switching capacity-with an
`50 MCI WorldCom frame relay network, even though they are
`similar when one takes the broader view that encompasses
`disparate networks like those discussed herein. The two
`frame relay providers have to agree upon information rates,
`switching capacities, frame sizes, etc. before the two net-
`55 works can communicate directly with each other.
`A configuration like that shown in FIG. 1 may be actively
`and routinely using both frame relay networks A and B. For
`instance, a local area network (LAN) at site 1 may be set up to
`send all traffic from the accounting and sales departments to
`60 router Al and send all traffic from the engineering department
`to router Bl. This may provide a very rough balance of the
`traffic load between the routers, but it does not attempt to
`balance router loads dynamically in response to actual traffic
`and thus is not "load-balancing" as that term is used herein.
`Alternatively, one of the frame relay networks may be a
`backup which is used only when the other frame relay net(cid:173)
`work becomes unavailable. In that case, it may take even
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 9 of 17
`
`

`

`US 7,406,048 B2
`
`3
`skilled network administrators several hours to perform the
`steps needed to switch the traffic away from the failed net(cid:173)
`work and onto the backup network, unless the invention of the
`'197 application is used. In general, the necessary Private
`Virtual Circuits (PVCs) must be established, routers at each
`site 102 must be reconfigured to use the correct serial links
`and PVCs, and LAN sat each site 102 must be reconfigured to
`point at the correct router as the default gateway.
`Although two private networks are shown in FIG. 1, three
`or more such networks could be employed, with similar con- 10
`siderations coming into play as to increased reliability, limits
`on load-balancing, the efforts needed to switch traffic when a
`network fails, and so on. Likewise, for clarity of illustration
`FIG. 1 shows only two sites, but three or more sites could
`communicate through one or more private networks.
`FIG. 2 illustrates a prior art configuration in which data is
`normally sent between sites 102 over a private network 106.
`A failover box 202 at each site 102 can detect failure of the
`network 106 and, in response to such a failure, will send the
`data instead over an ISDN link 204 while the network 106 is 20
`down.Using an ISDN link 204 as a backup is relatively easier
`and less expensive than using another private network 106 as
`the backup, but generally provides lower throughput. The
`ISDN link is an example of a point-to-point or leased line
`network link.
`FIG. 3 illustrates prior art configurations involving two
`private networks for increased reliability, in the sense that
`some of the sites in a given government agency or other entity
`302 can continue communicating even after one network goes
`down. For instance, if a frame relay network A goes down,
`sites 1, 2, and 3 will be unable to communicate with each
`other but sites 4, 5, and 6 will still be able to communicate
`amongst themselves through frame relay network B. Like(cid:173)
`wise, if network B goes down, sites 1, 2, and 3 will still be able
`to communicate through network A. Only if both networks go 35
`down at the same time would all sites be completely cut off.
`Like the FIG. 1 configurations, the FIG. 3 configuration uses
`two private networks. Unlike FIG. 1 however, there is no
`option for switching traffic to another private network when
`one network 106 goes down, although either or both of the
`networks in FIG. 3 could have an ISDN backup like that
`shown in FIG. 2. Note also that even when both private
`networks are up, sites 1, 2, and 3 communicate only among
`themselves; they are not connected to sites 4, 5, and 6. Net(cid:173)
`works A and Bin FIG. 3 are therefore not in "parallel" as that 45
`term is used herein, because all the traffic between each pair
`of sites goes through at most one of the networks A, B.
`FIG. 4 illustrates a prior art response to the incompatibility
`of frame relay networks of different carriers. A special "net(cid:173)
`work-to-network interface" (NNI) 402 is used to reliably 50
`transmit data between the two frame relay networks A and B.
`NNis are generally implemented in software at carrier offices.
`Note that the configuration in FIG. 4 does not provide addi(cid:173)
`tional reliability by using two frame relay networks 106,
`because those networks are in series rather than in parallel. If 55
`either of the frame relay networks A, Bin the FIG. 4 configu(cid:173)
`ration fails, there is no path between site 1 and site 2; adding
`the second frame relay network has not increased reliability.
`By contrast, FIG. 1 increases reliability by placing the frame
`relay networks in parallel, so that an alternate path is available 60
`if either (but not both) of the frame relay networks fails.
`Someone of skill in the art who was looking for ways to
`improve reliability by putting networks in parallel would
`probably not consider NNis pertinent, because they were
`used for serial configurations rather than parallel ones, and 65
`adding networks in a serial manner does not improve reliabil(cid:173)
`ity.
`
`4
`Internet-based communication solutions such as VPN sand
`Secure Sockets Layer (SSL) offer alternatives to frame relay
`106 and point-to-point leased line networks such as those
`using an ISDN link 204. These Internet-based solutions are
`advantageous in the flexibility and choice they offer in cost, in
`service providers, and in vendors. Accordingly, some organi(cid:173)
`zations have a frame relay 106 or leased line connection
`(a.k.a. point-to-point) for intranet communication and also
`have a connection for accessing the Internet 500, using an
`architecture such as that shown in FIG. 5.
`But better tools and techniques are needed for use in archi(cid:173)
`tectures such as that shown in FIG. 5. In particular, prior
`approaches for selecting which network to use for which
`packet(s) are coarse. For instance, all packets from depart-
`15 ment X might be sent over the frame relay connection 106
`while all packets from department Y are sent over the Internet
`500. Or the architecture might send all traffic over the frame
`relay network unless that network fails, and then be manually
`reconfigured to send all traffic over a VPN 502.
`Organizations are still looking for better ways to use Inter-
`net-based redundant connections to backup the primary
`frame relay networks. Also, organizations wanting to change
`from frame relay and point-to-point solutions to Internet(cid:173)
`based solutions have not had the option of transitioning in a
`25 staged manner. They have had to decide instead between the
`two solutions, and deploy the solution in their entire network
`communications system in one step. This is a barrier for
`deployment of Internet-based solutions 500/502, since an
`existing working network would be replaced by a yet-un-
`30 tested new network. Also, for organizations with several geo(cid:173)
`graphically distributed locations a single step conversion is
`very complex. Some organizations may want a redundant
`Internet-based backup between a few locations while main-
`taining the frame relay network for the entire organization.
`It would be an advancement in the art to provide new tools
`and techniques for configuring disparate networks (e.g.,
`frame relay/point-to-point WANs and Internet-based VPNs)
`in parallel, to obtain benefits such as greater reliability,
`improved security, and/or load-balancing. Such improve-
`40 ments are disclosed and claimed herein.
`
`BRIEF SUMMARY OF THE INVENTION
`
`The present invention provides tools and techniques for
`directing packets over multiple parallel disparate networks,
`based on addresses and other criteria. This helps organiza(cid:173)
`tions make better use of frame relay networks and/or point(cid:173)
`to-point (e.g., Tl, T3, fiber, OCx, Gigabit, wireless, or satel(cid:173)
`lite based) network connections in parallel with VPN s and/or
`other Internet-based networks. For instance, some embodi(cid:173)
`ments of the invention allow frame relay and VPN wide area
`networks to co-exist for redundancy as well as for transition(cid:173)
`ing from frame relay/point-to-point solutions to Internet(cid:173)
`based solutions in a staged manner. Some embodiments oper(cid:173)
`ate in configurations which communicate data packets over
`two or more disparate WAN connections, with the data traffic
`being dynamically load-balanced across the connections,
`while some embodiments treat one of the WANs as a backup
`for use mainly in case the primary connection through the
`other WAN fails.
`Other features and advantages of the invention will become
`more fully apparent through the following description.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`To illustrate the manner in which the advantages and fea(cid:173)
`tures of the invention are obtained, a more particular descrip-
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 10 of 17
`
`

`

`US 7,406,048 B2
`
`5
`tion of the invention will be given with reference to the
`attached drawings. These drawings only illustrate selected
`aspects of the invention and its context. In the drawings:
`FIG. 1 is a diagram illustrating a prior art approach having
`frame relay networks configured in parallel for increased
`reliability for all networked sites, in configurations that
`employ manual switchover between the two frame relay net(cid:173)
`works in case of failure.
`FIG. 2 is a diagram illustrating a prior art approach having
`a frame relay network configured in parallel with an ISDN 10
`network link for increased reliability for all networked sites.
`FIG. 3 is a diagram illustrating a prior art approach having
`independent and non-parallel frame relay networks, with
`each network connecting several sites but no routine or exten(cid:173)
`sive communication between the networks.
`FIG. 4 is a diagram illustrating a prior art approach having
`frame relay networks configured in series through a network(cid:173)
`to-network interface, with no consequent increase in reliabil-
`ity because the networks are in series rather than in parallel.
`FIG. 5 is a diagram illustrating a prior art approach having 20
`a frame relay network configured in parallel with a VPN or
`other Internet-based network that is disparate to the frame
`relay network, but without the fine-grained packet routing of
`the present invention.
`FIG. 6 is a diagram illustrating one system configuration of 25
`the present invention, in which the Internet and a private
`network are placed in parallel for increased reliability for all
`networked sites, without requiring manual traffic switchover,
`and with the option in some embodiments ofload balancing
`between the networks and/or increasing security by transmit- 30
`ting packets of a single logical connection over disparate
`networks.
`FIG. 7 is a diagram further illustrating a multiple disparate
`network access controller of the present invention, which
`comprises an interface component for each network to which 35
`the controller connects, and a path selector in the controller
`which uses one or more of the following as criteria: destina(cid:173)
`tion address, network status (up/down), network load, use of
`a particular network for previous packets in a given logical
`connection or session.
`FIG. 8 is a flowchart illustrating methods of the present
`invention for sending packets using a controller such as the
`one shown in FIG. 7.
`FIG. 9 is a flowchart illustrating methods of the present
`invention for combining connections to send traffic over mu!- 45
`tip le parallel independent disparate networks for reasons such
`as enhanced reliability, load balancing, and/or security.
`FIG. 10 is a diagram illustrating another system configu(cid:173)
`ration of the present invention, in which the Internet and a
`frame relay network are placed in parallel, with a VPN tunnel 50
`originating after the source controller and terminating before
`the destination controller, and each known site that is acces(cid:173)
`sible through one network is also accessible through the other
`network unless that other network fails.
`FIG. 11 is a diagram illustrating a system configuration
`similar to FIG. 10, except the VPN tunnel originates before
`the source controller and terminates after the destination con(cid:173)
`troller.
`
`55
`
`6
`tions, on the other hand. "Multiple" networks means two or
`more such networks. "Independent" means routing informa(cid:173)
`tion need not be shared between the networks. "Parallel" does
`not rule out all use of NNis and serial networks, but it does
`require that at least two of the networks in the configuration
`be in parallel at the location where the invention distributes
`traffic, so that alternate data paths through different networks
`are present. "Frame relay networks" or "private networks"
`does not rule out the use of an ISDN link or other backup for
`a particular frame relay or point-to-point private network, but
`it does require the presence of multiple such networks; FIG. 2,
`for instance, does not meet this requirement. A "frame relay
`network" is unavailable to the general public and thus dispar(cid:173)
`ate from the Internet and VPNs (which may be Internet-
`15 based), even though some traffic in the Internet may use
`public frame relay networks once the traffic leaves the loca(cid:173)
`tion where the invention distributes traffic.
`FIG. 6 illustrates one of many possible configurations of
`the present invention. Comments made here also apply to
`similar configurations involving only one or more frame relay
`networks 106, those involving only one or more point-to-
`point networks 204, and those not involving a VPN 604. for
`example. Two or more disparate networks are placed in par(cid:173)
`allel between two or more sites 102. In the illustrated con-
`figuration, the Internet 500 and a VPN 604 are disparate from,
`and in parallel with, frame relay/point-to-point network 106/
`204, with respect to site A and site B. No networks are parallel
`disparate networks in FIG. 6 with regard to site Casa traffic
`source, since that site is not connected to the Internet 500.
`Access to the disparate networks at site A and site Bis through
`an inventive controller 602 at each site. Additional controllers
`602 may be used at each location (i.e., controllers 602 may be
`placed in parallel to one another) in order to provide a
`switched connection system with no single point of failure.
`With continued attention to the illustrative network topol-
`ogy for one embodiment of the invention shown in FIG. 6, in
`this topology the three locations A, B, and Care connected to
`each other via a frame relay 106 or leased line network 204.
`Assume, for example, that all three locations are connected
`40 via a single frame relay network 106. Locations A and B are
`also connected to each other via a VPN connection 604. VPN
`tunnels are established between locations A and Bin the VPN,
`which pairs line 1 to line 3 and also pairs line 2 to line 3. There
`can be only one VPN tunnel between locations A and B. There
`is no VPN connection between location C and either location
`A or location B.
`Therefore, locations A, B, and C can communicate with
`each other over the frame relay network 106, and locations A
`and B (but not C) can also communicate with each other over
`the VPN connection 604. Communication between locations
`A and C, and communication between locations B and C, can
`take place over the frame relay network 106 only. Commu(cid:173)
`nication between locations A and B can take place over frame
`relay network 106. It can also take place over one of the lines
`l-and-3 pair, or the lines 2-and-3 pair, but not both atthe same
`time. Traffic can also travel over lines 2 and 4, but without a
`VPN tunnel. When the source and destination IP address pairs
`are the same between locations A and B but different types of
`networks connect those locations, as in FIG. 6 for instance,
`60 then a traffic routing decision that selects between network
`types cannot be made with an existing commercially avail(cid:173)
`able device. By contrast, the invention allows an organization
`to deploy an Internet-based solution between locations A and
`B while maintaining the frame relay network 106 between
`65 locations A, B, and C, and allows traffic routing that selects
`between the Internet and the frame relay network on a packet(cid:173)
`by-packet basis.
`
`DETAILED DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`
`The present invention relates to methods, systems, and
`configured storage media for connecting sites over multiple
`independent parallel disparate networks, such as frame relay
`networks and/or point-to-point network connections, on the
`one hand, and VPN s or other Internet-based network connec-
`
`Cisco Systems, Inc.
`Exhibit 1003
`Page 11 of 17
`
`

`

`US 7,406,048 B2
`
`8
`To operate as discussed herein, the invention uses informa(cid:173)
`tion about the IP address ranges in the locations reside as
`input data. For instance, a packet destined for the Internet 500
`is one whose destination address is not in any of the address
`ranges of the kuown locations (e.g., locations A, B, and C in
`the example of FIG. 6). In some configurations, this is the
`same as saying that a packet destined for the Internet is one
`whose address is not in the address range of any of the orga(cid:173)
`nization's locations. However, although all the kuown loca-
`10 tions may belong to a single organization, that is not a nec(cid:173)
`essary condition for using the invention. Known locations
`may also belong to multiple organizations or individuals.
`Likewise, other locations belonging to the organization may
`be unknown for purposes of a given embodiment of the inven-
`15 tion.
`Address ranges can be specified and tested by the control(cid:173)
`ler 602 using subnet masks. The subnet masks may be of
`different lengths (contain a different number of one bits) in
`different embodiments and/or in different address ranges in a
`20 given embodiment. For instance, class B and class C
`addresses may both be used in some embodiments.
`As another example, consider the illustrative network
`topology shown in FIG. 10. This configuration has two loca(cid:173)
`tions A and B which are connected by a frame relay network
`25 106 and by the Internet 500, through a frame relay router 105
`and an Internet router 104, at eac