`
`
`
`
`
`EXHIBIT
`
`EXHIBIT
`1001
`
`1001
`
`

`

`(12)
`
`United States Patent
`Datta et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,775,235 B2
`Aug. 10, 2004
`
`US006775235B2
`
`(54) TOOLS AND TECHNIQUES FOR
`[)[RECTING PACKETS ()VER [)ISPARATE
`NETWORKS
`
`(75) Inventors: Sanchaita Datta, Salt Lake City, UT
`(US)_ Ragula Bhaskar Salt Lake City
`’
`’
`’
`UT(US)
`
`_
`_
`(73) Asslgneei Ragula Systems, Salt Lake CRY, UT
`(Us)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`{gatsmct 553$???) grajjflusted under 35
`
`(21) Appl, No.1 10/361,837
`_
`(22) Flled:
`(65)
`
`Feb‘ 7’ 2003
`Prior Publication Data
`
`Us 2003/0147408 A1 Aug 7, 2003
`
`9/1999 Kitai et al. ............... .. 709/240
`5,948,069 A
`6,016,307 A * 1/2000 Kaplan et al. ...... ..
`370/238
`6,119,170 A * 9/2000 Schoffelman et al. .
`709/244
`6,128,298 A * 10/2000 Wootton et al. ..
`370/392
`6’253’247 B1
`6000* Bhaskar 6* a1‘
`709/237
`6,295,276 B1
`9/2001 Datta et al. ..... ..
`370/218
`6,339,595 B1
`1/2002 Rekhter et al.
`370/392
`6,438,100 B1
`8/2002 Halpern e161.
`370/218
`6,449,259 B1
`9/2002 Allain e161. .... ..
`370/253
`6,456,594 B1
`9/2002 Kaplan 6161.
`370/238
`6,493,341 B1
`12/2002 Datta e161.
`370/392
`6,493,349 B1
`12/2002 Casey ......... ..
`370/409
`6,665,702 B1 * 12/2003 Zisapel et al. ............ .. 718/105
`OTHER PUBLICATIONS
`‘RadWare announces LinkProof: The ?rst IP Load Balancing
`Solution for networks With multiple ISP connection’, Press
`Release, published Oct. 7, 1999.*
`‘RadWare Balances the Network’, Internet Traf?c Manage
`ment Center, published Jan. 1, 2000.* ~
`‘Global Product Spotlight: RadWare Lmkproof’, NetWork
`MagaZine.com, published Dec. 1, 1999.*
`
`Related US. Application Data
`
`(List Continued on IleXt page)
`
`(63) Continuation-in-part of application No. 10/034,197, ?led on
`Dec 28 2001
`(60) Provisional application No. 60/355,509, ?led on Feb. 8,
`2D0e(l~2,2z;nd28(r)(6visional application No. 60/259,269, ?led on
`
`elvm. Marcelo
`Primary Examiner
`(74) Attorney, Agent, or Fzrm—Thorpe North & Western
`LLP
`(57)
`
`ABSTRACT
`
`(51) Int. Cl.7 .............................................. .. H04L 12/64
`(52) us. Cl. ...................... .. 370/238; 370/252; 370/352
`(58) Field of Search ............................... .. 370/252 352
`370/230 235’ 238’
`’
`’
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,398,012 A
`3/1995 Derby et al. ......... .. 340/825.03
`5,420,862 A
`5/1995 Perlman _ _ _ _ _ _
`_ _ _ _ __ 370/85_13
`5,473,599 A 12/1995 Li 6161. ................. .. 370/16
`5,737,526 A
`4/1998 Periasamy et al.
`395/200.06
`5,898,673 A
`4/1999 Riggan et al. ............ .. 370/237
`
`Methods, Con?gured Storage media, and systems are PIO
`Vided for_<=ommunications Psing two or {more disparate
`networks 1n parallel to provlde load balancmg across net‘
`Work connections, greater reliability, and/or increased secu
`rity. A controller provides access to tWo or more disparate
`netWorks in parallel, through direct or indirect netWork
`interfaces. When one attached netWork fails, the failure is
`sensed by the controller and traffic is routed through one or
`more other disparate netWorks. When all attached disparate
`_
`netWorks are operatmg, one controller preferably balances
`the load between them
`
`24 Claims, 6 Drawing Sheets
`
`INTERNET @
`
`I
`
`Q
`Q
`6
`LINE 2
`LINE 1
`/—
`/_
`“g1
`8 ROUTER ROUTER
`,_ M M
`"zJ
`I
`|
`%
`VPN
`VPN
`'2 E Q
`_
`I
`SITE A CONTROLLER
`E _ E
`
`LINE 4
`LINE 3
`F
`/—
`ROUTER ROUTER
`193 M
`I
`VPN
`E
`I
`SITE B CONTROLLER
`E — E
`
`SITE C
`m
`
`'\
`ROUTER
`E
`I
`FRAME RELAY/ POINT-TO-POINT NETWORK 106/204
`
`LINE 5
`0:
`/—
`0 Lu %
`ROUTER
`z E o
`j 5' E E
`n: u.| LlJ
`LIJ 2 §
`5 IiI-I O I
`n: 0
`LI.
`
`LINE 7
`
`LINE 6
`
`—\
`ROUTER
`M
`
`I
`
`Viptela, Inc. - Exhibit 1001
`Page 1
`
`

`

`US 6,775,235 B2
`Page 2
`
`OTHER PUBLICATIONS
`
`‘RadWare Seeks Solutions to Easy—Access Problems’, South
`China Morning Post, published Dec. 7, 1999.*
`B. Gleeson et al., “A Framework for IP Based Virtual Private
`Networks,” RFC 2764 (Feb. 2000).
`US. patent application, Attorney Docket No. 3003.2.9A; see
`USPTO published application No. US—2002—0087724—A1,
`Jul. 4, 2002.
`T. Liao et al., “Using multiple links to interconnect LANs
`and public circuit sWitched data netWorks,” Proc. Int. Con
`ference on Communications Systems: Towards Global Inte
`gration, vol. 1, Singapore, 59 Nov. 1990, pp. 289—293.
`Press release from WWW.coyotepoint,corn, Sep. 8, 1997.
`NetWork Address Translation Technical Discussion, frorn
`safety.net; no later than May 7, 1999.
`Higginson et al., “Development of Router Clusters to Pro
`vide Fast Failover in IP Networks,” from WWW.asia—paci
`?c.digital.corn; no later than Sep. 29, 1998.
`
`Pages from WWW.navpoint.corn; no later than Dec. 24, 2001.
`“The Basic Guide to Frame Relay NetWorking”, pp. 1—85,
`copyright date 1998.
`“NNI & UNI”, pp. 1—2, Nov. 16, 2001.
`“Disaster Recovery for Frame Relay NetWorks”, pp. 1—14,
`no later than Dec. 7, 2001.
`T. Nolle, “Watching Your Back”, pp. 1—3, Nov. 1, 1999.
`“Multi—Attached and Multi—Horned Dedicated Access”, pp.
`1—5, no later than Dec. 8, 2001.
`Feibel, “InternetWork Link,” Novell’s® Cornplete Encyclo
`pedia of NetWorking, copyright date 1995.
`Tanenbaurn, Computer Networks (3rd Ed.), pp. 396—406;
`copyright date 1996.
`WeXler, “Frame Relay and IP VPNs: Cornpete Or CoeXist?”,
`frorn WWW.bcr.corn; Jul. 1999.
`
`* cited by eXarniner
`
`Viptela, Inc. - Exhibit 1001
`Page 2
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 1 0f 6
`
`US 6,775,235 B2
`
`ROUTER A1
`E
`
`SITE 1
`E
`
`ROUTER B1
`19.52
`
`FRAME RELAY
`NETWORK A
`1%
`
`FRAME RELAY
`NETWORK B
`£3.52
`
`ROUTER A2
`1%
`
`SITE 2
`E
`
`ROUTER B2
`lQE
`
`(PRIOR ART)
`Fig. 1
`
`SITE 1
`E
`
`ROUTER 1 E
`FAILOVER
`COMPONENT Q
`
`FRAME RELAY
`NETWORK 1_O6_
`
`ISDN NETWORK
`LINK @
`
`ROUTER 2 @
`
`FAILOVER
`COMPONENT E SITE 2
`m
`
`(PRIOR ART)
`Fig. 2
`
`Viptela, Inc. - Exhibit 1001
`Page 3
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 2 0f 6
`
`US 6,775,235 B2
`
`CORPORATION OR OTHER ENTITY Q
`
`SITE 1
`E
`
`SITE 4
`Q
`
`FRAME RELAY
`NETWORK A
`E
`
`FRAME RELAY
`NETWORK B
`E
`
`SITE 5
`SITE 3
`SITE 2
`Q Q IQ
`
`SITE 6
`19.2.
`
`(PRIOR ART)
`Fig. 3
`
`SITE 1
`102
`
`ROUTER 1
`E
`I
`FRAME RELAY
`NETWORK A E
`I
`N ETWO RK-TO-N ETWORK
`INTERFACE 5Q
`
`FRAME RELAY
`NETWORK B JQQ
`I
`ROUTER 2
`E5.
`
`(PRIOR ART)
`Fig. 4
`
`SITE 2
`1_0_2
`
`Viptela, Inc. - Exhibit 1001
`Page 4
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 3 0f 6
`
`US 6,775,235 B2
`
`ROUTER A1
`1%
`
`SITE 1
`1%
`
`ROUTER B1
`M
`
`FRAME RELAY
`NETWORK A
`E
`
`INTERNET / VIRTUAL
`PRIVATE NETWORK
`M22
`
`ROUTER A2
`E
`
`SITE 2
`E
`
`ROUTER B2
`1%’.
`
`(PRIOR ART)
`Fig. 5
`
`SITE E
`
`MULTIPLE DISPARATE NETWORK ACCESS
`CONTROLLER §0_2_
`
`SITE INTERFACE m2
`PACKET PATH SELECTOR (E.G., LOAD
`BALANCING, REDUNDANCY, SECURITY) ZO_4
`INTERFACE
`INTERFACE
`INTERFACE
`Q5
`7_06
`m
`
`TO A
`NETWORK
`BY PATH
`A1
`
`TO A
`NETWORK
`BY PATH
`A2
`
`TO A
`NETWORK
`BY PATH
`A3
`
`Fig. 7
`
`Viptela, Inc. - Exhibit 1001
`Page 5
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 4 0f 6
`
`US 6,775,235 B2
`
`INTERNET 5%
`
`/— LINE 1 /— LINE 2
`
`‘2
`9
`E3 L21
`Z
`8 ROUTER ROUTER
`L m E
`E
`l
`VPN
`%
`VPN
`I; E w
`
`/— LINE 3 f LINE4
`
`ROUTER ROUTER
`E m
`I
`VPN
`w
`
`SITE A CONTROLLER
`E _ Q92
`
`SITE B CONTROLLER
`E w E
`
`SITE C
`E
`
`LINE 6
`
`_\
`ROUTER
`E
`
`LINE 7
`
`T
`ROUTER
`E
`
`LINE 5
`1
`/—
`0 Lu %
`ROUTER
`<>£ g Q
`—' _' E E
`LIJ O Q
`C! UJ Lu
`LIJ ‘2 §
`5 5 O
`E
`0
`
`FRAME RELAY / POINT-TO-POINT NETWORK 106/204
`
`Fig. 6
`INTERNET @
`
`l
`I
`ROUTER z
`ROUTER x
`E
`E
`\
`l
`CONTROLLER SITE B
`SITE A CONTROLLER
`B Q2 _ E
`E _ A 5%
`I
`l
`ROUTER W
`ROUTER Y
`E
`E
`I
`l
`FRAME RELAY NETWORK 1%
`Fig. 10
`
`Viptela, Inc. - Exhibit 1001
`Page 6
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 5 0f 6
`
`US 6,775,235 B2
`
`—>
`
`SPECIFY PATH SELECTOR CRITERIA §_Q_Q
`>I<
`SEND PACKET(S) TO CONTROLLER Q
`
`<—
`
`DETECT NETWORK FAILURE @
`I
`ROUTE AROUND FAILURE @
`
`Fig. 8
`
`I
`
`I
`
`OBTAIN ADDRESS % OBTAIN SYSTEM
`RANGE
`¢~—————I
`TOPOLOGY
`INFORMATION w
`INFORMATION QQZ
`I
`I
`
`w
`
`RECEIVE PACKET FROM LOCAL SITE %
`I
`I
`LOOK FOR ADDRESS TO “KNOWN” DESTINATION @
`I
`SELECT PATH TO A DISPARATE NETWORK %
`USE LOAD BALANCING CRITERION g9
`
`USE CONNECTIVITY CRITERION _91_2
`
`USE SECURITY CRITERION _91_4
`
`I
`MODIFY PACKET DESTINATION ADDRESS 916
`l
`FORWARD PACKET ON SELECTED PATH gig
`I
`Fig. 9
`
`<———
`
`Viptela, Inc. - Exhibit 1001
`Page 7
`
`

`

`U.S. Patent
`
`Aug. 10, 2004
`
`Sheet 6 6f 6
`
`US 6,775,235 B2
`
`INTERNET _50_0
`
`l
`l
`ROUTER Z
`ROUTER X
`£5
`M
`l
`I
`VPN A CONTROLLER CONTROLLER VPN B
`191
`A @Q
`5 L2
`lQl
`|
`I
`I
`I
`SITE A
`ROUTER Y
`ROUTER W
`SITE B
`E
`E
`E
`E
`I
`FRAME RELAY NETWORK @
`Fig. 11
`
`Viptela, Inc. - Exhibit 1001
`Page 8
`
`

`

`US 6,775,235 B2
`
`1
`TOOLS AND TECHNIQUES FOR
`DIRECTING PACKETS OVER DISPARATE
`NETWORKS
`
`RELATED APPLICATIONS
`
`This application claims priority to commonly owned
`copending US. provisional patent application serial No.
`60/355,509 ?led Feb. 8, 2002, which is also incorporated
`herein by reference. This application is a continuation-in
`part of US. patent application Ser. No. 10/034,197 ?led
`Dec. 28, 2001, which claims priority to US. provisional
`patent application serial No. 60/259,269 ?led Dec. 29, 2000,
`each of which is also incorporated herein by reference.
`
`FIELD OF THE INVENTION
`
`The present invention relates to computer network data
`transmission, and more particularly relates to tools and
`techniques for communications using disparate parallel
`networks, such as a virtual private network (“VPN”) or the
`Internet in parallel with a point-to-point, leased line, or
`frame relay network, in order to help provide bene?ts such
`as load balancing across network connections, greater
`reliability, and increased security.
`
`TECHNICAL BACKGROUND OF THE
`INVENTION
`
`Organizations have used frame relay networks and point
`to-point leased line networks for interconnecting geographi
`cally dispersed of?ces or locations. These networks have
`been implemented in the past and are currently in use for
`interoffice communication, data exchange and ?le sharing.
`Such networks have advantages, some of which are noted
`below. But these networks also tend to be expensive, and
`there are relatively few options for reliability and redun
`dancy. As networked data communication becomes critical
`to the day-to-day operation and functioning of an
`organiZation, the need for lower cost alternatives for redun
`dant back-up for wide area networks becomes important.
`Frame relay networking technology offers relatively high
`throughput and reliability. Data is sent in variable length
`frames, which are a type of packet. Each frame has an
`address that the frame relay network uses to determine the
`frame’s destination. The frames travel to their destination
`through a series of switches in the frame relay network,
`which is sometimes called a network “cloud”; frame relay is
`an eXample of packet-switched networking technology. The
`transmission lines in the frame relay cloud must be essen
`tially error-free for frame relay to perform well, although
`error handling by other mechanisms at the data source and
`destination can compensate to some eXtent for lower line
`reliability. Frame relay and/or point-to-point network ser
`vices are provided or have been provided by various carriers,
`such as AT&T, Qwest, X0, and MCI WorldCom.
`Frame relay networks are an eXample of a network that is
`“disparate” from the Internet and from Internet-based virtual
`private networks for purposes of the present invention.
`Another eXample of such a “disparate” network is a point
`to-point network, such as a T1 or T3 connection. Although
`the underlying technologies differ somewhat, for purposes
`of the present invention frame relay networks and point-to
`point networks are generally equivalent in important ways,
`such as the conventional reliance on manual switchovers
`when traf?c must be redirected after a connection fails, and
`their implementation distinct from the Internet. A frame
`relay permanent virtual circuit is a virtual point-to-point
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`connection. Frame relays are used as eXamples throughout
`this document, but the teachings will also be understood in
`the conteXt of point-to-point networks.
`A frame relay or point-to-point network may become
`suddenly unavailable for use. For instance, both MCI World
`Com and AT &T users have lost access to their respective
`frame relay networks during major outages. During each
`outage, the entire network failed. Loss of a particular line or
`node in a network is relatively easy to work around. But loss
`of an entire network creates much larger problems.
`Tools and techniques to permit continued data transmis
`sion after loss of an entire frame relay network that would
`normally carry data are discussed in US. patent application
`Ser. No. 10/034,197 ?led Dec. 28, 2001 and incorporated
`herein. The ’197 application focuses on architectures involv
`ing two or more “private” networks in parallel, whereas the
`present application focuses on architectures involving dis
`parate networks in parallel, such as a proprietary frame relay
`network and the Internet. Note that the term “private net
`wor ” is used herein in a manner consistent with its use in
`the ’197 application (which comprises frame relay and
`point-to-point networks), eXcept that a “virtual private net
`work” as discussed herein is not a “private network”. Virtual
`private networks are Internet-based, and hence disparate
`from private networks, i.e., from frame relay and point-to
`point networks. To reduce the risk of confusion that might
`arise from misunderstanding “private network” to comprise
`“virtual private networ ” herein, virtual private networks
`will be henceforth referred to as VPNs. Other differences
`and similarities between the present application and the ’197
`application will also be apparent to those of skill in the art
`on reading the two applications.
`Various architectures involving multiple networks are
`known in the art. For instance, FIG. 1 illustrates prior art
`con?gurations involving two frame relay networks for
`increased reliability; similar con?gurations involve one or
`more point-to-point network connections. Two sites 102
`transmit data to each other (alternately, one site might be
`only a data source, while the other is only a data destination).
`Each site has two border routers 105. Two frame relay
`networks 106, 108 are available to the sites 102 through the
`routers 105. The two frame relay networks 106, 108 have
`been given separate numbers in the ?gure, even though each
`is a frame relay network, to emphasiZe the incompatibility of
`frame relay networks provided by different carriers. An
`AT&T frame relay network, for instance, is incompatible—
`in details such as maXimum frame siZe or switching
`capacity—with an MCI WorldCom frame relay network,
`even though they are similar when one takes the broader
`view that encompasses disparate networks like those dis
`cussed herein. The two frame relay providers have to agree
`upon information rates, switching capacities, frame siZes,
`etc. before the two networks can communicate directly with
`each other.
`Acon?guration like that shown in FIG. 1 may be actively
`and routinely using both frame relay networks A and B. For
`instance, a local area network (LAN) at site 1 may be set up
`to send all traf?c from the accounting and sales departments
`to router A1 and send all traf?c from the engineering
`department to router B1. This may provide a very rough
`balance of the traffic load between the routers, but it does not
`attempt to balance router loads dynamically in response to
`actual traffic and thus is not “load-balancing” as that term is
`used herein.
`Alternatively, one of the frame relay networks may be a
`backup which is used only when the other frame relay
`
`Viptela, Inc. - Exhibit 1001
`Page 9
`
`

`

`US 6,775,235 B2
`
`10
`
`15
`
`25
`
`35
`
`3
`network becomes unavailable. In that case, it may take even
`skilled network administrators several hours to perform the
`steps needed to switch the traffic away from the failed
`network and onto the backup network, unless the invention
`of the ’197 application is used. In general, the necessary
`Private Virtual Circuits (PVCs) must be established, routers
`at each site 102 must be recon?gured to use the correct serial
`links and PVCs, and LANs at each site 102 must be
`recon?gured to point at the correct router as the default
`gateway.
`Although two private networks are shown in FIG. 1, three
`or more such networks could be employed, with similar
`considerations coming into play as to increased reliability,
`limits on load-balancing, the efforts needed to switch traf?c
`when a network fails, and so on. Likewise, for clarity of
`illustration FIG. 1 shows only two sites, but three or more
`sites could communicate through one or more private net
`works.
`FIG. 2 illustrates a prior art con?guration in which data is
`normally sent between sites 102 over a private network 106.
`A failover box 202 at each site 102 can detect failure of the
`network 106 and, in response to such a failure, will send the
`data instead over an ISDN link 204 while the network 106
`is down. Using an ISDN link 204 as a backup is relatively
`easier and less expensive than using another private network
`106 as the backup, but generally provides lower throughput.
`The ISDN link is an example of a point-to-point or leased
`line network link.
`FIG. 3 illustrates prior art con?gurations involving two
`private networks for increased reliability, in the sense that
`some of the sites in a given government agency or other
`entity 302 can continue communicating even after one
`network goes down. For instance, if a frame relay network
`A goes down, sites 1, 2, and 3 will be unable to communicate
`with each other but sites 4, 5, and 6 will still be able to
`communicate amongst themselves through frame relay net
`work B. Likewise, if network B goes down, sites 1, 2, and
`3 will still be able to communicate through network A. Only
`if both networks go down at the same time would all sites be
`completely cut off. Like the FIG. 1 con?gurations, the FIG.
`3 con?guration uses two private networks. Unlike FIG. 1,
`however, there is no option for switching traf?c to another
`private network when one network 106 goes down, although
`either or both of the networks in FIG. 3 could have an ISDN
`backup like that shown in FIG. 2. Note also that even when
`both private networks are up, sites 1, 2, and 3 communicate
`only among themselves; they are not connected to sites 4, 5,
`and 6. Networks A and B in FIG. 3 are therefore not in
`“parallel” as that term is used herein, because all the traf?c
`between each pair of sites goes through at most one of the
`networks A, B.
`FIG. 4 illustrates a prior art response to the incompatibil
`ity of frame relay networks of different carriers. A special
`“network-to-network interface” (NNI) 402 is used to reli
`ably transmit data between the two frame relay networks A
`55
`and B. NNIs are generally implemented in software at
`carrier of?ces. Note that the con?guration in FIG. 4 does not
`provide additional reliability by using two frame relay
`networks 106, because those networks are in series rather
`than in parallel. If either of the frame relay networks A, B in
`the FIG. 4 con?guration fails, there is no path between site
`1 and site 2; adding the second frame relay network has not
`increased reliability. By contrast, FIG. 1 increases reliability
`by placing the frame relay networks in parallel, so that an
`alternate path is available if either (but not both) of the frame
`relay networks fails. Someone of skill in the art who was
`looking for ways to improve reliability by putting networks
`
`40
`
`45
`
`65
`
`4
`in parallel would probably not consider NNIs pertinent,
`because they were used for serial con?gurations rather than
`parallel ones, and adding networks in a serial manner does
`not improve reliability.
`Internet-based communication solutions such as VPNs
`and Secure Sockets Layer (SSL) offer alternatives to frame
`relay 106 and point-to-point leased line networks such as
`those using an ISDN link 204. These Internet-based solu
`tions are advantageous in the ?exibility and choice they offer
`in cost, in service providers, and in vendors. Accordingly,
`some organiZations have a frame relay 106 or leased line
`connection (a.k.a. point-to-point) for intranet communica
`tion and also have a connection for accessing the Internet
`500, using an architecture such as that shown in FIG. 5.
`But better tools and techniques are needed for use in
`architectures such as that shown in FIG. 5. In particular,
`prior approaches for selecting which network to use for
`which packet(s) are coarse. For instance, all packets from
`department X might be sent over the frame relay connection
`106 while all packets from department Y are sent over the
`Internet 500. Or the architecture might send all traffic over
`the frame relay network unless that network fails, and then
`be manually recon?gured to send all traf?c over a VPN 502.
`Organizations are still looking for better ways to use
`Internet-based redundant connections to backup the primary
`frame relay networks. Also, organiZations wanting to change
`from frame relay and point-to-point solutions to Internet
`based solutions have not had the option of transitioning in a
`staged manner. They have had to decide instead between the
`two solutions, and deploy the solution in their entire network
`communications system in one step. This is a barrier for
`deployment of Internet-based solutions 500/502, since an
`existing working network would be replaced by a yet
`untested new network. Also, for organiZations with several
`geographically distributed locations a single step conversion
`is very complex. Some organiZations may want a redundant
`Internet-based backup between a few locations while main
`taining the frame relay network for the entire organiZation.
`It would be an advancement in the art to provide new tools
`and techniques for con?guring disparate networks (e.g.,
`frame relay/point-to-point WANs and Internet-based VPNs)
`in parallel, to obtain bene?ts such as greater reliability,
`improved security, and/or load-balancing. Such improve
`ments are disclosed and claimed herein.
`
`BRIEF SUMMARY OF THE INVENTION
`The present invention provides tools and techniques for
`directing packets over multiple parallel disparate networks,
`based on addresses and other criteria. This helps organiZa
`tions make better use of frame relay networks and/or point
`to-point (e.g., T1, T3, ?ber, OCx, Gigabit, wireless, or
`satellite based) network connections in parallel with VPNs
`and/or other Internet-based networks. For instance, some
`embodiments of the invention allow frame relay and VPN
`wide area networks to co-exist for redundancy as well as for
`transitioning from frame relay/point-to-point solutions to
`Internet-based solutions in a staged manner. Some embodi
`ments operate in con?gurations which communicate data
`packets over two or more disparate WAN connections, with
`the data traf?c being dynamically load-balanced across the
`connections, while some embodiments treat one of the
`WANs as a backup for use mainly in case the primary
`connection through the other WAN fails.
`Other features and advantages of the invention will
`become more fully apparent through the following descrip
`tion.
`
`Viptela, Inc. - Exhibit 1001
`Page 10
`
`

`

`US 6,775,235 B2
`
`15
`
`25
`
`5
`BRIEF DESCRIPTION OF THE DRAWINGS
`To illustrate the manner in which the advantages and
`features of the invention are obtained, a more particular
`description of the invention will be given with reference to
`the attached drawings. These drawings only illustrate
`selected aspects of the invention and its context. In the
`drawings:
`FIG. 1 is a diagram illustrating a prior art approach having
`frame relay networks con?gured in parallel for increased
`reliability for all networked sites, in con?gurations that
`employ manual switchover between the two frame relay
`networks in case of failure.
`FIG. 2 is a diagram illustrating a prior art approach having
`a frame relay network con?gured in parallel with an ISDN
`network link for increased reliability for all networked sites.
`FIG. 3 is a diagram illustrating a prior art approach having
`independent and non-parallel frame relay networks, with
`each network connecting several sites but no routine or
`extensive communication between the networks.
`FIG. 4 is a diagram illustrating a prior art approach having
`frame relay networks con?gured in series through a
`network-to-network interface, with no consequent increase
`in reliability because the networks are in series rather than
`in parallel.
`FIG. 5 is a diagram illustrating a prior art approach having
`a frame relay network con?gured in parallel with a VPN or
`other Internet-based network that is disparate to the frame
`relay network, but without the ?ne-grained packet routing of
`the present invention.
`FIG. 6 is a diagram illustrating one system con?guration
`of the present invention, in which the Internet and a private
`network are placed in parallel for increased reliability for all
`networked sites, without requiring manual traf?c
`switchover, and with the option in some embodiments of
`load balancing between the networks and/or increasing
`security by transmitting packets of a single logical connec
`tion over disparate networks.
`FIG. 7 is a diagram further illustrating a multiple disparate
`network access controller of the present invention, which
`comprises an interface component for each network to
`which the controller connects, and a path selector in the
`controller which uses one or more of the following as
`criteria: destination address, network status (up/down), net
`work load, use of a particular network for previous packets
`in a given logical connection or session.
`FIG. 8 is a ?owchart illustrating methods of the present
`invention for sending packets using a controller such as the
`one shown in FIG. 7.
`FIG. 9 is a ?owchart illustrating methods of the present
`invention for combining connections to send traf?c over
`multiple parallel independent disparate networks for reasons
`such as enhanced reliability, load balancing, and/or security.
`FIG. 10 is a diagram illustrating another system con?gu
`ration of the present invention, in which the Internet and a
`frame relay network are placed in parallel, with a VPN
`tunnel originating after the source controller and terminating
`55
`before the destination controller, and each known site that is
`accessible through one network is also accessible through
`the other network unless that other network fails.
`FIG. 11 is a diagram illustrating a system con?guration
`similar to FIG. 10, eXcept the VPN tunnel originates before
`the source controller and terminates after the destination
`controller.
`
`45
`
`35
`
`40
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`The present invention relates to methods, systems, and
`con?gured storage media for connecting sites over multiple
`
`65
`
`6
`independent parallel disparate networks, such as frame relay
`networks and/or point-to-point network connections, on the
`one hand, and VPNs or other Internet-based network
`connections, on the other hand. “Multiple” networks means
`two or more such networks. “Independent” means routing
`information need not be shared between the networks.
`“Parallel” does not rule out all use of NNIs and serial
`networks, but it does require that at least two of the networks
`in the con?guration be in parallel at the location where the
`invention distributes traf?c, so that alternate data paths
`through different networks are present. “Frame relay net
`works” or “private networks” does not rule out the use of an
`ISDN link or other backup for a particular frame relay or
`point-to-point private network, but it does require the pres
`ence of multiple such networks; FIG. 2, for instance, does
`not meet this requirement. A “frame relay network” is
`unavailable to the general public and thus disparate from the
`Internet and VPNs (which may be Internet-based), even
`though some traf?c in the Internet may use public frame
`relay networks once the traf?c leaves the location where the
`invention distributes traf?c.
`FIG. 6 illustrates one of many possible con?gurations of
`the present invention. Comments made here also apply to
`similar con?gurations involving only one or more frame
`relay networks 106, those involving only one or more
`point-to-point networks 204, and those not involving a VPN
`604, for eXample. Two or more disparate networks are
`placed in parallel between two or more sites 102. In the
`illustrated con?guration, the Internet 500 and a VPN 604 are
`disparate from, and in parallel with, frame relay/point-to
`point network 106/204, with respect to site A and site B. No
`networks are parallel disparate networks in FIG. 6 with
`regard to site C as a traffic source, since that site is not
`connected to the Internet 500. Access to the disparate
`networks at site A and and site B is through an inventive
`controller 602 at each site. Additional controllers 602 may
`be used at each location (i.e., controllers 602 may be placed
`in parallel to one another) in order to provide a switched
`connection system with no single point of failure.
`With continued attention to the illustrative network topol
`ogy for one embodiment of the invention shown in FIG. 6,
`in this topology the three locations A, B, and C are con
`nected to each other via a frame relay 106 or leased line
`network 204. Assume, for eXample, that all three locations
`are connected via a single frame relay network 106. Loca
`tions A and B are also connected to each other via a VPN
`connection 604. VPN tunnels are established between loca
`tions A and B in the VPN, which pairs line 1 to line 3 and
`also pairs line 2 to line 3. There can be only one VPN tunnel
`between locations A and B. There is no VPN connection
`between location C and either location A or location B.
`Therefore, locations A, B, and C can communicate with
`each other over the frame relay network 106, and locations
`A and B (but not C) can also communicate with each other
`over the VPN connection 604. Communication between
`locations A and C, and communication between locations B
`and C, can take place over the frame relay network 106 only.
`Communication between locations A and B can take place
`over frame relay network 106. It can also take place over one
`of the lines 1-and-3 pair, or the lines 2-and-3 pair, but not
`both at the same time. Traffic can also travel over lines 2 and
`4, but without a VPN tunnel. When the source and destina
`tion IP address pairs are the same between locations A and
`B but different types of networks connect those locations, as
`in FIG. 6 for instance, then a traffic routing decision that
`selects between network types cannot be made with an
`eXisting commercially available device. By contrast, the
`
`Viptela, Inc. - Exhibit 1001
`Page 11
`
`

`

`US 6,775,235 B2
`
`7
`invention allows an organization to deploy an Internet-based
`solution between locations A and B While maintaining the
`frame relay netWork 106 betWeen locations A, B, and C, and
`alloWs traf?c routing that selects betWeen the Internet and
`the frame relay netWork on a packet-by-packet basis.
`The invention may thus be con?gured to alloW the orga
`niZation to achieve the folloWing goals, in the context of
`FIG. 6; similar goals are facilitated in other con?gurations.
`First, the organiZation can deploy an Internet-based second
`connection betWeen only locations A and B, While main
`taining frame relay connectivity betWeen locations A, B, and
`C. Later the organiZation may deploy an Internet-based
`solution at location C as Well. Second, the organiZation can
`use the Internet-based connection betWeen locations A and B
`for full load-balancing or backup, or a combination of the
`tWo. Third, the organiZation can use the frame relay con
`nection betWeen locations A and B for full load-balancing or
`backup, or a combination of the tWo. Fourth, the organiZa
`tion can load-balance traf?c in a multi-homing situation
`betWeen tWo ISPs or tWo connections to the Internet at
`locations A and/or B.
`To better understand the invention, consider the operation
`of controller device 602 at location A. The controller 602
`examines the IP data traf?c meant to go through it and makes
`determinations and takes steps such as those discussed
`beloW.
`If the traffic is destined for the Internet 500, send the
`traf?c over the Internet using lines 1 and/or 2. Load balanc
`ing decisions that guide the controller 602 in distri