EXHIBIT
`
`EXHIBIT
`1003
`
`1003
`
`

`
`US007406048B2
`
`(12) Ulllted States Patent
`Datta et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 7,406,048 B2
`Jul. 29, 2008
`
`(54) TOOLS AND TECHNIQUES FOR DIRECTING
`PACKETS OVER DISPARATE NETWORKS
`
`(76) Inventors: Sanchaita Datta, 4540 S. Jupiter Dr.,
`Salt Lake City, UT (Us) 84124; Bhaskar
`
`6,016,307 A *
`6,253,247 B1
`6,295,276 B1
`6,339,595 B1
`6,438,100 B1
`
`1/2000 Kaplan etal. ............. .. 370/238
`6/2001 Bhaskar et a1.
`9/2001 Datta et a1.
`1/2002 Rekhter et a1,
`8/2002 Halpern et a1.
`
`
`
`Juplter Dr.’ C113” UT (Us) 84124
`
`Lake
`
`6,456,594 B1
`
`
`
`et 9/2002 Kaplan et a1.
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`.
`.
`patent is extended or adjusted under 35
`U'S'C~ 154(1)) by 821 days'
`
`6’493’341 B1
`6,493,349 B1
`*
`7,088,716 B2
`2002/0087724 A1
`
`12/2002 Dana et 31'
`12/2002 Casey
`_
`8/2006 Sugal et al. ............... .. 370/392
`7/2002 Datta et a1.
`
`(21) App1.N0.: 10/911,846
`
`(22) Filed:
`
`Aug. 3, 2004
`
`(65)
`
`Prior Publication Data
`
`Us 2005/0008017 A1
`
`Jan- 13, 2005
`
`Related US. Application Data
`(63) Continuation-in-part of application No. 10/034,197,
`?led on Dec. 28, 2001.
`_
`_
`_
`_
`
`(60) 55031380011211 appl1cat1on No. 60/259,269, ?led on Dec.
`
`(51) Int Cl
`(2006.01)
`H04L 12/64
`(52) US. Cl. ...... .... ...... ... ..... .. 370/238; 370/252; 370/352
`(58) Field of Classi?cation Search ............... .. 370/238,
`S
`1_
`_
`?l f
`1
`1113170052’ 352
`ee app lcanon e or Comp ete Seam lstory'
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`3/1995 D b t l.
`5 398 012 A
`er y e a
`’
`’
`5/1995 Perlman
`5,420,862 A
`5,473,599 A 121995 Li et 31‘
`5,737,526 A
`4/1998 Periasamy et al.
`5,898,673 A
`4/1999 Riggan et a1.
`5,948,069 A
`9/1999 Kitai et al.
`
`OTHER PUBLICATIONS
`
`T. Liao et al., “Using multiple links to interconnect LANs and public
`circuit switched data networks,” Proc. Int. Conference on Commu
`nications Systems: Towards Global Integration, vol. 1, Singapore, 59
`Nov. 1990, pp. 289-293.
`
`(Continued)
`
`Primary ExamineriMelvin Marcelo
`
`(74) Attorney Agent] or FI-FmAOgiIVie Law Firm
`
`(57)
`
`ABSTRACT
`
`Methods, Con?gured storage media, and Systems are prO_
`Vided for Communications using two or more disparate Hep
`works in parallel to provide load balancing across network
`connections, greater reliability, and/or increased security. A
`controller provides access to two or more disparate networks
`in parallel, through direct or indirect network interfaces.
`When one attached network fails, the failure is sensed by the
`controller and tra?ic is routed through one or more other
`.
`.
`d1sparate networks. When all attached d1 sparate networks are
`.
`operating, one controller preferably balances the load
`between them
`
`24 Claims, 6 Drawing Sheets
`
`l
`
`INTERNET @
`
`% L
`5
`LIN
`LINE 1
`/— E2
`/—
`%
`8 ROUTER ROUTER
`,_ M M
`"2‘
`|
`|
`5
`VPN
`VPN
`E % %
`
`/—L|NE3 /—L|NE4
`ROUTER ROUTER
`lo_4 M
`|
`VPN
`@A
`
`SITE A _‘ CONTROLLER
`12
`.52
`
`SITE B J CONTROLLER
`E w
`
`SITE C
`Q
`
`Do: Lu 0)
`Q E 5
`u-j 5‘ I:
`o
`0c LLI |_|_|
`UJ 2 §
`55 8 |
`
`LI.
`
`/— LINE 5
`ROUTER
`1_Q§
`
`LINE 6 —\
`ROUTER
`@
`
`LINE 7 —\
`ROUTER
`105
`
`_
`
`FRAME RELAY / POINT-TO-POINT NETWORK 106/204
`
`1
`
`Viptela, Inc. - Exhibit 1003
`Page 1
`
`

`
`US 7,406,048 B2
`Page 2
`
`OTHER PUBLICATIONS
`Press release from www.coyotepoint.com, Sep. 8, 1997.
`Network Address Translation Technical Discussion, from safety.net;
`no later than May 7, 1999.
`Higginson et al., “Development of Router Clusters to Provide Fast
`Failover in IP Networks,” from www.asia-paci?c.digital.com; no
`later than Sep. 29, 1998.
`Pages from www.navpoint.com; no later than Dec. 24, 2001.
`“The Basic Guide to Frame Relay Networking”, pp. 1-85, copyright
`date 1998.
`“NNI & UNI”, pp. 1-2, Nov. 16, 2001.
`“Disaster Recovery for Frame Relay Networks”, pp. 1-14, no later
`than Dec. 7, 2001.
`
`T. Nolle, “WatchingYour Back”, pp. 1-3, Nov. 1, 1999.
`Multi-Attached and Multi-Homed Dedicated Accessz, pp. 1-5, no
`later than Dec. 8, 2001.
`Feibel, “Internetwork Link,” Novel’s® Complete Encyclopedia of
`Networking, copyright date 1995.
`Tanenbaum, Computer Networks (3rd Ed.), pp. 396-406; copyright
`date 1996.
`WeXler, “Frame Relay and IP VPNs: Complete Ore CoeXist?”, from
`www.bcr.com; Jul. 1999.
`B. Gleeson et al., “A Framework for IP Based Virtual Private Net
`works,” RFC 2764 (Feb. 2000).
`
`* cited by examiner
`
`Viptela, Inc. - Exhibit 1003
`Page 2
`
`

`
`US. Patent
`
`Jul. 29, 2008
`
`Sheet 1 of6
`
`US 7,406,048 B2
`
`ROUTER A1
`Di
`
`SITE 1
`m;
`
`ROUTER B1
`E
`
`FRAME RELAY
`NETWORK A
`E
`
`FRAME RELAY
`NETWORK B
`ii
`
`ROUTER A2
`E5
`
`SITE 2
`E2.
`
`ROUTER B2
`1%
`
`(PRIOR ART)
`Fig. 1
`
`SITE 1
`J2
`
`ROUTER 1 E
`FAILOVER
`COMPONENT Q
`
`FRAME RELAY
`NETWORK 1o_e
`
`ISDN NETWORK
`LINK 2g
`
`ROUTER 2 E
`
`FAILOVER
`COMPONENT 2%
`
`(PRIOR ART)
`Fig. 2
`
`SITE 2
`1i
`
`Viptela, Inc. - Exhibit 1003
`Page 3
`
`

`
`US. Patent
`
`Jul. 29,2008
`
`Sheet 2 0f 6
`
`US 7,406,048 B2
`
`CORPORATION OR OTHER ENTITY &
`
`SITE 1
`1E
`
`SITE 4
`.12
`
`FRAME RELAY
`NETWORK A
`1&
`
`FRAME RELAY
`NETWORK B
`195.
`
`SITE 2
`E
`
`SITE 3
`.LL
`
`SITE 5
`1_0_2_
`
`SITE 6
`M
`
`(PRIOR ART)
`Fig. 3
`
`SITE 1
`LQZ
`
`ROUTER 1
`
`FRAME RELAY
`NETWORK A E
`
`NETWORK-TO-NETWORK
`INTERFACE fLQZ
`
`FRAME RELAY
`NETWORK B 1%
`I
`ROUTER 2
`Q5
`
`(PRIOR ART)
`Fig. 4
`
`SITE 2
`Q
`
`Viptela, Inc. - Exhibit 1003
`Page 4
`
`

`
`US. Patent
`
`Jul. 29, 2008
`
`Sheet 3 of6
`
`US 7,406,048 B2
`
`ROUTER A1
`195
`
`SITE 1
`E
`
`ROUTER B1
`M
`
`FRAME RELAY
`NETWORK A
`l9§
`
`INTERNET / VIRTUAL
`PRIVATE NETWORK
`M2
`
`ROUTER A2
`@Q
`
`SITE 2
`12
`
`ROUTER 82
`1'3
`
`(PRIOR ART)
`Fig. 5
`
`SITE
`1%
`
`MULTIPLE DISPARATE NETWORK ACCESS
`CONTROLLER Q
`
`SITE INTERFACE _70_2
`PACKET PATH SELECTOR (E.G., LOAD
`BALANCING, REDUNDANCY, SECURITY) 7_O4_
`INTERFACE
`INTERFACE
`INTERFACE
`Z26.
`7_Q§
`Z95.
`
`TO A
`NETWORK
`BY PATH
`A1
`
`TO A
`NETWORK
`BY PATH
`A2
`
`TO A
`NETWORK
`BY PATH
`A3
`
`Fig. 7
`
`Viptela, Inc. - Exhibit 1003
`Page 5
`
`

`
`US. Patent
`
`Jul. 29, 2008
`
`Sheet 4 of6
`
`US 7,406,048 B2
`
`INTERNET @
`
`Q
`9
`5
`LIN 1
`/— E /—LlNE2
`g
`O ROUTER ROUTER
`,9 m m
`Li’
`I
`I
`35
`VPN
`VPN
`5
`§_0_4
`6i
`
`LIN 4
`/—L|NE3 /— E
`ROUTER ROUTER
`M 193
`I
`VPN
`@1
`
`SITE A CONTROLLER
`.12 '—
`QB
`
`SITE 8 CONTROLLER
`m _ w
`
`SITE C
`E
`
`% w
`UJ
`2 g 5
`—' _' I:
`UJ O O
`[I UJ Lu
`LLI 2 %
`E 51 O
`E 0
`
`/— LINE 5
`ROUTER
`E
`
`LINE 6 -\
`ROUTER
`E
`
`LINE 7 —\
`ROUTER
`1_0_5
`
`FRAME RELAY / POINT-TO-POINT NETWORK 106/204
`
`Fig. 6
`INTERNET Q
`
`I
`I
`ROUTER z
`ROUTER x
`M
`M
`I
`I
`CONTROLLER sITE B
`SITE A CONTROLLER
`B E _ E
`1% _ A63
`I
`I
`ROUTER w
`ROUTER Y
`M
`E
`I
`I
`FRAME RELAY NETWORK E
`Fig. 10
`
`Viptela, Inc. - Exhibit 1003
`Page 6
`
`

`
`US. Patent
`
`Jul. 29, 2008
`
`Sheet 5 of6
`
`US 7,406,048 B2
`
`—>
`
`SPECIFY PATH SELECTOR CRITERIA §O_0
`
`A
`
`SEND PACKET(S) TO CONTROLLER Q12
`
`r
`DETECT NETWORK FAILURE @
`I
`ROUTE AROUND FAILURE 80
`
`Fig. 8
`
`I
`
`I
`
`OBTAIN ADDRESS __> OBTAIN SYSTEM
`RANGE
`<_‘ TOPOLOGY
`INFORMATION QIE
`INFORMATION Q
`v
`I
`
`>I
`RECEIVE PACKET FROM LOCAL SITE %
`L
`I
`LOOK FOR ADDRESS TO “KNOWN" DESTINATION 2%
`I
`SELECT PATH TO A DISPARATE NETWORK 9 8
`USE LOAD BALANCING CRITERION 9_1Q
`
`USE CONNECTIVITY CRITERION 2Q
`
`USE SECURITY CRITERION 9_14_
`I
`MODIFY PACKET DESTINATION ADDRESS 9_1_(_5_
`I
`I
`FORWARD PACKET ON SELECTED PATH w
`I
`I
`Fig. 9
`
`Viptela, Inc. - Exhibit 1003
`Page 7
`
`

`
`US. Patent
`
`Jul. 29, 2008
`
`Sheet 6 of6
`
`US 7,406,048 B2
`
`INTERNET _5_QQ
`
`I
`l
`ROUTER z
`ROUTER x
`M
`M
`I
`I
`VPN A CONTROLLER CONTROLLER VPN B
`M _ A Q2
`B 6M
`M
`l
`1
`l
`|
`SITE A
`ROUTER Y
`ROUTER W
`SITE B
`1-0-2-
`M
`M
`M
`l
`|
`FRAME RELAY NETWORK JQQ
`Fig. 11
`
`Viptela, Inc. - Exhibit 1003
`Page 8
`
`

`
`US 7,406,048 B2
`
`1
`TOOLS AND TECHNIQUES FOR DIRECTING
`PACKETS OVER DISPARATE NETWORKS
`
`RELATED APPLICATIONS
`
`This application is a continuation of forthcoming US. Pat.
`No. 6,775,235 (application Ser. No. 10/361,837), which
`claims priority to commonly owned US. provisional patent
`application Ser. No. 60/355,509 ?led Feb. 8,2002, which
`provisional application is also incorporated herein by refer
`ence. US. Pat. No. 6,775,235 is a continuation-in-part ofU.S.
`patent application Ser. No. 10/034,197 ?led Dec. 28, 2001,
`which claims priority to US. provisional patent application
`Ser. No. 60/259,269 ?led Dec. 29, 2000, each ofwhich appli
`cations is also incorporated herein by reference.
`
`FIELD OF THE INVENTION
`
`The present invention relates to computer network data
`transmission, and more particularly relates to tools and tech
`niques for communications using disparate parallel networks,
`such as a virtual private network (“VPN”) or the Internet in
`parallel with a point-to-point, leased line, or frame relay
`network, in order to help provide bene?ts such as load bal
`ancing across network connections, greater reliability, and
`increased security.
`
`TECHNICAL BACKGROUND OF THE
`INVENTION
`
`Organizations have used frame relay networks and point
`to-point leased line networks for interconnecting geographi
`cally dispersed of?ces or locations. These networks have
`been implemented in the past and are currently in use for
`interof?ce communication, data exchange and ?le sharing.
`Such networks have advantages, some of which are noted
`below. But these networks also tend to be expensive, and there
`are relatively few options for reliability and redundancy. As
`networked data communication becomes critical to the day
`to-day operation and functioning of an organiZation, the need
`for lower cost alternatives for redundant back-up for wide
`area networks becomes important.
`Frame relay networking technology offers relatively high
`throughput and reliability. Data is sent in variable length
`frames, which are a type of packet. Each frame has an address
`that the frame relay network uses to determine the frame’s
`destination. The frames travel to their destination through a
`series of switches in the frame relay network, which is some
`times called a network “cloud”; frame relay is an example of
`packet-switched networking technology. The transmission
`lines in the frame relay cloud must be essentially error-free
`for frame relay to perform well, although error handling by
`other mechanisms at the data source and destination can
`compensate to some extent for lower line reliability. Frame
`relay and/or point-to-point network services are provided or
`have been provided by various carriers, such as AT&T, Qwest,
`X0, and MCI WorldCom.
`Frame relay networks are an example of a network that is
`“disparate” from the Internet and from Intemet-based virtual
`private networks for purposes of the present invention.
`Another example of such a “disparate” network is a point-to
`point network, such as a T1 or T3 connection. Although the
`underlying technologies differ somewhat, for purposes of the
`present invention frame relay networks and point-to-point
`networks are generally equivalent in important ways, such as
`the conventional reliance on manual switchovers when tra?ic
`must be redirected after a connection fails, and their imple
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`mentation distinct from the Internet. A frame relay permanent
`virtual circuit is a virtual point-to-point connection. Frame
`relays are used as examples throughout this document, but the
`teachings will also be understood in the context of point-to
`point networks.
`A frame relay or point-to-point network may become sud
`denly unavailable for use. For instance, both MCI WorldCom
`and AT&T users have lost access to their respective frame
`relay networks during major outages. During each outage, the
`entire network failed. Loss of a particular line or node in a
`network is relatively easy to work around. But loss of an entire
`network creates much larger problems.
`Tools and techniques to permit continued data transmis
`sion after loss of an entire frame relay network that would
`normally carry data are discussed in US. patent application
`Ser. No. 10/034,197 ?led Dec. 28, 2001 and incorporated
`herein. The ’ 197 application focuses on architectures involv
`ing two or more “private” networks in parallel, whereas the
`present application focuses on architectures involving dispar
`ate networks in parallel, such as a proprietary frame relay
`network and the Internet. Note that the term “private net
`work” is used herein in a manner consistent with its use in the
`’197 application (which comprises frame relay and point-to
`point networks), except that a “virtual private network” as
`discussed herein is not a “private network”. Virtual private
`networks are Internet-based, and hence disparate from private
`networks, i.e., from frame relay and point-to -point networks.
`To reduce the risk of confusion that might arise from misun
`derstanding “private networ ” to comprise “virtual private
`network” herein, virtual private networks will be henceforth
`referred to as VPNs. Other differences and similarities
`between the present application and the ’197 application will
`also be apparent to those of skill in the art on reading the two
`applications.
`Various architectures involving multiple networks are
`known in the art. For instance, FIG. 1 illustrates prior art
`con?gurations involving two frame relay networks for
`increased reliability; similar con?gurations involve one or
`more point-to-point network connections. Two sites 102
`transmit data to each other (alternately, one site might be only
`a data source, while the other is only a data destination). Each
`site has two border routers 105. Two frame relay networks
`106, 108 are available to the sites 102 through the routers 105.
`The two frame relay networks 106, 108 have been given
`separate numbers in the ?gure, even though each is a frame
`relay network, to emphasize the incompatibility of frame
`relay networks provided by different carriers. An AT&T
`frame relay network, for instance, is incompatibleiin details
`such as maximum frame siZe or switching capacityiwith an
`MCI WorldCom frame relay network, even though they are
`similar when one takes the broader view that encompasses
`disparate networks like those discussed herein. The two
`frame relay providers have to agree upon information rates,
`switching capacities, frame siZes, etc. before the two net
`works can communicate directly with each other.
`A con?guration like that shown in FIG. 1 may be actively
`and routinely using both frame relay networks A and B. For
`instance, a local area network (LAN) at site 1 may be set up to
`send all tra?ic from the accounting and sales departments to
`routerAl and send all traf?c from the engineering department
`to router B1. This may provide a very rough balance of the
`traf?c load between the routers, but it does not attempt to
`balance router loads dynamically in response to actual traf?c
`and thus is not “load-balancing” as that term is used herein.
`Alternatively, one of the frame relay networks may be a
`backup which is used only when the other frame relay net
`work becomes unavailable. In that case, it may take even
`
`Viptela, Inc. - Exhibit 1003
`Page 9
`
`

`
`US 7,406,048 B2
`
`3
`skilled network administrators several hours to perform the
`steps needed to switch the traf?c away from the failed net
`work and onto the backup network, unless the invention of the
`’l97 application is used. In general, the necessary Private
`Virtual Circuits (PVCs) must be established, routers at each
`site 102 must be recon?gured to use the correct serial links
`and PVCs, and LANs at each site 102 must be recon?gured to
`point at the correct router as the default gateway.
`Although two private networks are shown in FIG. 1, three
`or more such networks could be employed, with similar con
`siderations coming into play as to increased reliability, limits
`on load-balancing, the efforts needed to switch tra?ic when a
`network fails, and so on. Likewise, for clarity of illustration
`FIG. 1 shows only two sites, but three or more sites could
`communicate through one or more private networks.
`FIG. 2 illustrates a prior art con?guration in which data is
`normally sent between sites 102 over a private network 106.
`A failover box 202 at each site 102 can detect failure of the
`network 106 and, in response to such a failure, will send the
`data instead over an ISDN link 204 while the network 106 is
`down. Using an ISDN link 204 as a backup is relatively easier
`and less expensive than using another private network 106 as
`the backup, but generally provides lower throughput. The
`ISDN link is an example of a point-to-point or leased line
`network link.
`FIG. 3 illustrates prior art con?gurations involving two
`private networks for increased reliability, in the sense that
`some of the sites in a given government agency or other entity
`302 can continue communicating even after one network goes
`down. For instance, if a frame relay network A goes down,
`sites 1, 2, and 3 will be unable to communicate with each
`other but sites 4, 5, and 6 will still be able to communicate
`amongst themselves through frame relay network B. Like
`wise, if network B goes down, sites 1, 2, and 3 will still be able
`to communicate through networkA. Only if both networks go
`down at the same time would all sites be completely cut off.
`Like the FIG. 1 con?gurations, the FIG. 3 con?guration uses
`two private networks. Unlike FIG. 1 however, there is no
`option for switching tra?ic to another private network when
`one network 106 goes down, although either or both of the
`networks in FIG. 3 could have an ISDN backup like that
`shown in FIG. 2. Note also that even when both private
`networks are up, sites 1, 2, and 3 communicate only among
`themselves; they are not connected to sites 4, 5, and 6. Net
`works A and B in FIG. 3 are therefore not in “parallel” as that
`term is used herein, because all the tra?ic between each pair
`of sites goes through at most one of the networks A, B.
`FIG. 4 illustrates a prior art response to the incompatibility
`of frame relay networks of different carriers. A special “net
`work-to-network interface” (NN I) 402 is used to reliably
`transmit data between the two frame relay networks A and B.
`NNIs are generally implemented in software at carrier o?ices.
`Note that the con?guration in FIG. 4 does not provide addi
`tional reliability by using two frame relay networks 106,
`because those networks are in series rather than in parallel. If
`either of the frame relay networks A, B in the FIG. 4 con?gu
`ration fails, there is no path between site 1 and site 2; adding
`the second frame relay network has not increased reliability.
`By contrast, FIG. 1 increases reliability by placing the frame
`relay networks in parallel, so that an alternate path is available
`if either (but not both) of the frame relay networks fails.
`Someone of skill in the art who was looking for ways to
`improve reliability by putting networks in parallel would
`probably not consider NNIs pertinent, because they were
`used for serial con?gurations rather than parallel ones, and
`adding networks in a serial manner does not improve reliabil
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`Internet-based communication solutions such asVPNs and
`Secure Sockets Layer (SSL) offer alternatives to frame relay
`106 and point-to-point leased line networks such as those
`using an ISDN link 204. These Internet-based solutions are
`advantageous in the ?exibility and choice they offer in cost, in
`service providers, and in vendors. Accordingly, some organi
`Zations have a frame relay 106 or leased line connection
`(a.k.a. point-to-point) for intranet communication and also
`have a connection for accessing the Internet 500, using an
`architecture such as that shown in FIG. 5.
`But better tools and techniques are needed for use in archi
`tectures such as that shown in FIG. 5. In particular, prior
`approaches for selecting which network to use for which
`packet(s) are coarse. For instance, all packets from depart
`ment X might be sent over the frame relay connection 106
`while all packets from departmentY are sent over the Internet
`500. Or the architecture might send all tra?ic over the frame
`relay network unless that network fails, and then be manually
`recon?gured to send all tra?ic over a VPN 502.
`Organizations are still looking for better ways to use Inter
`net-based redundant connections to backup the primary
`frame relay networks. Also, organiZations wanting to change
`from frame relay and point-to-point solutions to Intemet
`based solutions have not had the option of transitioning in a
`staged manner. They have had to decide instead between the
`two solutions, and deploy the solution in their entire network
`communications system in one step. This is a barrier for
`deployment of Internet-based solutions 500/502, since an
`existing working network would be replaced by a yet-un
`tested new network. Also, for organiZations with several geo
`graphically distributed locations a single step conversion is
`very complex. Some organizations may want a redundant
`Internet-based backup between a few locations while main
`taining the frame relay network for the entire organiZation.
`It would be an advancement in the art to provide new tools
`and techniques for con?guring disparate networks (e.g.,
`frame relay/point-to-point WANs and Internet-based VPNs)
`in parallel, to obtain bene?ts such as greater reliability,
`improved security, and/or load-balancing. Such improve
`ments are disclosed and claimed herein.
`
`BRIEF SUMMARY OF THE INVENTION
`
`The present invention provides tools and techniques for
`directing packets over multiple parallel disparate networks,
`based on addresses and other criteria. This helps organiZa
`tions make better use of frame relay networks and/or point
`to-point (e.g., T1, T3, ?ber, OCx, Gigabit, wireless, or satel
`lite based) network connections in parallel with VPNs and/or
`other Internet-based networks. For instance, some embodi
`ments of the invention allow frame relay and VPN wide area
`networks to co-exist for redundancy as well as for transition
`ing from frame relay/point-to-point solutions to Intemet
`based solutions in a staged manner. Some embodiments oper
`ate in con?gurations which communicate data packets over
`two or more disparate WAN connections, with the data traf?c
`being dynamically load-balanced across the connections,
`while some embodiments treat one of the WANs as a backup
`for use mainly in case the primary connection through the
`other WAN fails.
`Other features and advantages of the invention will become
`more fully apparent through the following description.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`To illustrate the manner in which the advantages and fea
`tures of the invention are obtained, a more particular descrip
`
`Viptela, Inc. - Exhibit 1003
`Page 10
`
`

`
`US 7,406,048 B2
`
`5
`tion of the invention will be given with reference to the
`attached drawings. These drawings only illustrate selected
`aspects of the invention and its context. In the drawings:
`FIG. 1 is a diagram illustrating a prior art approach having
`frame relay networks con?gured in parallel for increased
`reliability for all networked sites, in con?gurations that
`employ manual switchover between the two frame relay net
`works in case of failure.
`FIG. 2 is a diagram illustrating a prior art approach having
`a frame relay network con?gured in parallel with an ISDN
`network link for increased reliability for all networked sites.
`FIG. 3 is a diagram illustrating a prior art approach having
`independent and non-parallel frame relay networks, with
`each network connecting several sites but no routine or exten
`sive communication between the networks.
`FIG. 4 is a diagram illustrating a prior art approach having
`frame relay networks con?gured in series through a network
`to-network interface, with no consequent increase in reliabil
`ity because the networks are in series rather than in parallel.
`FIG. 5 is a diagram illustrating a prior art approach having
`a frame relay network con?gured in parallel with a VPN or
`other Internet-based network that is disparate to the frame
`relay network, but without the ?ne-grained packet routing of
`the present invention.
`FIG. 6 is a diagram illustrating one system con?guration of
`the present invention, in which the Internet and a private
`network are placed in parallel for increased reliability for all
`networked sites, without requiring manual traf?c switchover,
`and with the option in some embodiments of load balancing
`between the networks and/ or increasing security by transmit
`ting packets of a single logical connection over disparate
`networks.
`FIG. 7 is a diagram further illustrating a multiple disparate
`network access controller of the present invention, which
`comprises an interface component for each network to which
`the controller connects, and a path selector in the controller
`which uses one or more of the following as criteria: destina
`tion address, network status (up/down), network load, use of
`a particular network for previous packets in a given logical
`connection or session.
`FIG. 8 is a ?owchart illustrating methods of the present
`invention for sending packets using a controller such as the
`one shown in FIG. 7.
`FIG. 9 is a ?owchart illustrating methods of the present
`invention for combining connections to send tra?ic over mul
`tiple parallel independent disparate networks for reasons such
`as enhanced reliability, load balancing, and/or security.
`FIG. 10 is a diagram illustrating another system con?gu
`ration of the present invention, in which the Internet and a
`frame relay network are placed in parallel, with a VPN tunnel
`originating after the source controller and terminating before
`the destination controller, and each known site that is acces
`sible through one network is also accessible through the other
`network unless that other network fails.
`FIG. 11 is a diagram illustrating a system con?guration
`similar to FIG. 10, except the VPN tunnel originates before
`the source controller and terminates after the destination con
`troller.
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`DETAILED DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`
`60
`
`The present invention relates to methods, systems, and
`con?gured storage media for connecting sites over multiple
`independent parallel disparate networks, such as frame relay
`networks and/or point-to-point network connections, on the
`one hand, and VPNs or other Internet-based network connec
`
`65
`
`6
`tions, on the other hand. “Multiple” networks means two or
`more such networks. “Independent” means routing informa
`tion need not be shared between the networks. “Parallel” does
`not rule out all use of NNIs and serial networks, but it does
`require that at least two of the networks in the con?guration
`be in parallel at the location where the invention distributes
`traf?c, so that alternate data paths through different networks
`are present. “Frame relay networks” or “private networks”
`does not rule out the use of an ISDN link or other backup for
`a particular frame relay or point-to -point private network, but
`it does require the presence of multiple such networks; FIG. 2,
`for instance, does not meet this requirement. A “frame relay
`networ ” is unavailable to the general public and thus dispar
`ate from the Internet and VPNs (which may be Intemet
`based), even though some tra?ic in the Internet may use
`public frame relay networks once the traf?c leaves the loca
`tion where the invention distributes traf?c.
`FIG. 6 illustrates one of many possible con?gurations of
`the present invention. Comments made here also apply to
`similar con?gurations involving only one or more frame relay
`networks 106, those involving only one or more point-to
`point networks 204, and those not involving a VPN 604. for
`example. Two or more disparate networks are placed in par
`allel between two or more sites 102. In the illustrated con
`?guration, the Internet 500 and aVPN 604 are disparate from,
`and in parallel with, frame relay/point-to-point network 106/
`204, with respect to siteA and site B. No networks are parallel
`disparate networks in FIG. 6 with regard to site C as a traf?c
`source, since that site is not connected to the Internet 500.
`Access to the disparate networks at siteA and site B is through
`an inventive controller 602 at each site. Additional controllers
`602 may be used at each location (i.e., controllers 602 may be
`placed in parallel to one another) in order to provide a
`switched connection system with no single point of failure.
`With continued attention to the illustrative network topol
`ogy for one embodiment of the invention shown in FIG. 6, in
`this topology the three locations A, B, and C are connected to
`each other via a frame relay 106 or leased line network 204.
`Assume, for example, that all three locations are connected
`via a single frame relay network 106. Locations A and B are
`also connected to each other via a VPN connection 604. VPN
`tunnels are establishedbetween locationsA and B in theVPN,
`which pairs line 1 to line 3 and also pairs line 2 to line 3. There
`can be only one VPN tunnel between locations A and B. There
`is no VPN connection between location C and either location
`A or location B.
`Therefore, locations A, B, and C can communicate with
`each other over the frame relay network 106, and locations A
`and B (but not C) can also communicate with each other over
`the VPN connection 604. Communication between locations
`A and C, and communication between locations B and C, can
`take place over the frame relay network 106 only. Commu
`nication between locations A and B can take place over frame
`relay network 106. It can also take place over one of the lines
`1-and-3 pair, or the lines 2-and-3 pair, but not both at the same
`time. Tra?ic can also travel over lines 2 and 4, but without a
`VPN tunnel. When the source and destination IP address pairs
`are the same between locations A and B but different types of
`networks connect those locations, as in FIG. 6 for instance,
`then a tra?ic routing decision that selects between network
`types cannot be made with an existing commercially avail
`able device. By contrast, the invention allows an organization
`to deploy an Internet-based solution between locations A and
`B while maintaining the frame relay network 106 between
`locations A, B, and C, and allows traf?c routing that selects
`between the Internet and the frame relay network on a packet
`by-packet basis.
`
`Viptela, Inc. - Exhibit 1003
`Page 11
`
`

`
`US 7,406,048 B2
`
`7
`The invention may thus be con?gured to allow the organi
`Zation to achieve the following goals, in the context of FIG. 6;
`similar goals are facilitated in other con?gurations. First, the
`organiZation can deploy an Intemet-based second connection
`betWeen only locations A and B, While maintaining frame
`relay connectivity betWeen locations A, B, and C. Later the
`organiZation may deploy an Internet-based solution at loca
`tion C as Well. Second, the organization can use the Intemet
`based connection betWeen locations A and B for full load
`balancing or backup, or a combination of the tWo. Third, the
`organiZation can use the frame relay connection betWeen
`locations A and B for full load-balancing or backup, or a
`combination of the tWo. Fourth, the organiZation can load
`balance tra?ic in a multi-homing situation betWeen tWo ISPs
`or tWo connections to the Internet at locations A and/ or B.
`To better understand the invention, consider the operation
`of controller device 602 at location A. The controller 602
`examines the IP data tra?ic meant to go through it and makes
`determinations and takes steps such as those discussed beloW.
`If the tra?ic is destined for the Internet 500, send the tra?ic
`over the Internet using lines 1 and/or 2. Load balancing deci
`sions that guide the controller 602 in distributing packets
`betWeen the lines can be based on criteria such as the load of
`a given netWork, router, or connection relative to other net
`Works, routers, or connections, to be performed dynamically
`in response to actual traf?c. Load-balancing may be done
`through a round-robin algorithm Which places the next TCP
`or UDP session on the next available line, or it may involve
`more complex algorithms that attempt to measure and track
`the throughput, latency, and/or other performance character
`istics of a given link or path element. Load-balancing is
`preferably done on a per-packet basis for site-to-site data
`tra?ic over the Internet or frame relay net, or done on a TCP
`or UDP session basis for Internet tra?ic, as opposed to prior
`approaches that use a per-department and/or per-router basis
`for dividing traf?c. Load-balancing algorithms in gene