US009264301B1
`
`a2) United States Patent
`Chua et al.
`
`(10) Patent No.:
`
`US 9,264,301 Bl
`
`(45) Date of Patent:
`
`*Feb. 16, 2016
`
`(54) HIGH AVAILABILITY FOR SOFTWARE
`
`(56) References Cited
`
`DEFINED NETWORKS
`
`(71) Applicant: Wiretap Ventures, LLC, Santa Clara,
`
`CA (US)
`
`(72) Inventors: Roy Liang Chua, Saratoga, CA (US);
`
`Matthew Palmer, Menlo Park, CA
`
`(US); Andrew Keith Pearce, San
`
`Francisco, CA (US); David Warren
`
`Hawley, Belmont, CA (US)
`
`(73)
`
`Assignee: Wiretap Ventures, LLC, Santa Clara,
`
`CA (US)
`
`Notice: Subject to any disclaimer, the term of this
`
`patent is extended or adjusted under 35
`
`U.S.C. 154(b) by 38 days.
`
`This patent is subject to a terminal dis-
`
`claimer.
`
`(21)
`
`(22)
`
`Appl. No.: 13/841,968
`
`Filed: Mar. 15, 2013
`
`U.S. PATENT DOCUMENTS
`
`6,954,775 Bl
`
`7,024,600 B2
`
`7,342,890 B1*
`7,519,294 B2*
`
`7,607,049 B2
`
`7,796,601 Bl
`
`7,801,031 B2*
`7,813,263 B2*
`7,969,862 B1*
`
`8,369,213 B2*
`
`8,396,950 Bl
`
`8,619,546 B2
`
`8,755,389 Bl
`
`8,755,893 B2*
`8,811,212 B2*
`
`10/2005
`4/2006
`
`3/2008
`4/2009
`10/2009
`9/2010
`
`9/2010
`10/2010
`6/2011
`
`2/2013
`3/2013
`12/2013
`
`6/2014
`6/2014
`8/2014
`
`Shanklin et al.
`Manley et al.
`
`Ferguson vo. 370/238
`Bullock wc. 398/59
`Yasuie et al.
`
`Norman
`
`Chao et al. oo... 370/228
`Chang etal. oo. 370/216
`Wang etal. oe 370/216
`
`Vasseur etal. oc... 370/228
`Sandick et al.
`Nandagopal et al.
`
`Poutievski et al.
`Gross etal. oe 607/46
`Beheshti-Zavareh
`
`2001/0045914 Al
`2002/0152320 Al*
`
`etal. 370/252
`11/2001 Bunker
`10/2002 Lau wees 709/238
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`U.S. Appl. No. 13/842,067, by Matthew Palmer, filed Mar. 15, 2013.
`
`(Continued)
`
`Related U.S. Application Data
`
`Primary Examiner — Phuongchau Ba Nguyen
`
`(60) Provisional application No. 61/703,692, filed on Sep.
`
`(74) Attorney, Agent, or Firm — Pillsbury Winthrop Shaw
`Pittman LLP
`
`20, 2012.
`
`(51)
`
`Int. Cl.
`
`GOIR 31/08
`
`HOAL 12/24
`
`HO4L 12/701
`
`(2006.01)
`
`(2006.01)
`
`(2013.01)
`
`(52)
`
`U.S. Cl.
`
`CPC ceceeseeees HOAL 41/0668 (2013.01); HO4L 45/00
`
`(2013.01)
`
`(58)
`
`Field of Classification Search
`
`(57) ABSTRACT
`
`In one example, a controller device for a software defined
`network (SDN) includes one or more network interfaces con-
`
`figured to communicate with network devices of the SDN,
`and one or more processors configured to determine a pri-
`mary path for network traffic between network devices of the
`
`SDN, determine a backup path to the primary path for the
`
`network traffic between the network devices of the SDN, and,
`
`in response to a failover event, configure the network devices
`
`USPC wee 370/252, 254, 238; 709/220, 221;
`
`to switch from the primary path to the backup path.
`
`714/4.2, E11.008
`
`See application file for complete search history.
`
`20 Claims, 9 Drawing Sheets
`
`SDN CONTROLLER
`iz
`
`CONTROL UNIT
`130
`
`PATH PATH SERVICE
`SOU NT | [DETERMINATION] | VERIFICATION DEVICE
`12 UNIT UNIT CONTROL UNIT
`134 136 ii
`
`ON MODEL
`
`$ Mo RULES INFORMATION
`
`
`
`SECURITY
`
`148
`
`7 ry
`
`q ¥
`
`USER
`INToeekcE SDN INTERFACE
`
`150
`
`

`

`US 9,264,301 Bl
`
`Page 2
`
`(56) References Cited 2013/0044636 Al 2/2013. Koponen
`2013/0058208 Al 3/2013 Pfaff et al.
`U.S. PATENT DOCUMENTS 2013/0163475 Al 6/2013. Beliveau et al.
`2013/0173810 Al 7/2013 Subramaniam
`
`2004/0039827 Al 9/9004. Thomas 2013/022344? Al 8/2013 Narayanan et al.
`2005/0138111 Al 6/2005 Aton etal. 2013/0275552 Al 10/2013 Dhesikan et al.
`2005/0265228 Al* 12/2005 Fredette et al. ou... 370/216 2013/0283374 Al 10/2013 Zisapel et al.
`2006/0092857 Al §/2006 Ansari etal. 2013/0286893 Al* 10/2013 Zhuetal. oe 370/254
`2007/0005740 Al 1/2007 DiFalco et al. 2013/0329601 Al* 12/2013 Yinetal. wc. 370/254
`2007/0088845 Al 4/2007 Memon etal. 2013/0332982 Al — 12/2013 Rao etal.
`2007/0115967 Al 5/2007 Vandenberghe 2013/0333029 Al = 12/2013 Chesla et al.
`2007/0220175 Al* 9/2007 Khanna etal... 709/251 2014/0003232 Al — 1/2014 Guichard et al.
`2008/0062871 Al 3/2008 Grayson 2014/0020072 Al 1/2014 Thomas
`2008/0219156 AL* 9/2008 Caviglia et al. ccc... 370/228 2014/0052830 Al 2/2014 Bigall et al.
`2008/0232347 AL* 9/2008 Chaoet al. ccc 370/351 2014/0052836 Al = 2/2014 Nguyen et al.
`2009/0201803 Al* 8/2009 Filsfils etal. ccc... 370/222 2014/0169158 Al* 6/2014 Mishra etal. ................ 370/228
`2009/0249115 Al* 10/2009 Bycroft et al. oi.ccccccce 714/4 2014/0195666 Al = 7/2014 Dumitriu et al.
`2010/0054122 Al* 3/2010 Tayloretal. ccc. 370/225 2015/0023210 Al 1/2015 Kis
`2010/0266279 Al* 10/2010 Sadananda ou... 398/48
`2010/0293043 A1* 11/2010 Atreyaet al... 705/14.4 OTHER PUBLICATIONS
`2011/0196953 Al 8/2011 Samaha
`2011/0264795 Al 10/2011 Koide et al. U.S. Appl. No. 13/842,064, by Matthew Palmer, filed Mar. 15, 2013.
`
`2011/0286324 Al 11/2011 Bellagamba et al. U.S. Appl. No. 13/842,192, by Matthew Palmer, filed Mar. 15, 2013.
`2012/0044801 Al* 2/2012 Vasseur etal. 0.00.5 370/217 U.S. Appl. No. 13/842,186, by Matthew Palmer, filed Mar. 15, 2013.
`2012/0176890 Al : 7/2012 Balus et al... eee 370/218 U.S. Appl. No. 13/842,264, by Matthew Palmer, filed Mar. 15, 2013.
`2012/0195186 Al 8/2012 Singh et al. dceeeceesceecueeeas 370/217 McKeown et al., “OpenFlow: Enabling Innovation in Campus Net-
`2012/0227091 Al 9/2012 Smith ks" C C ‘cation Review, ACM SIGCOMM. vol
`9012/0287791 Al 11/2012 Xi et al. WOTKS, omputer ommunication eview, , ¥OI.
`2013/0010600 Al 1/2013 Jocha et al. 38(2), Apr. 2008, 6 pp.
`2013/0028070 Al*® 1/2013 Beheshti-Zavareh U.S. Appl. No. 61/625,872, by Yehuda Zisapel, filed Apr. 18, 2012.
`
`a 370/217
`2013/0028073 Al* 1/2013 Tatipamula et al. .......... 370/218 * cited by examiner
`
`

`

`VOLADIAAG YSAANAS
`MYOMLAN
`O1TASIAAG
`a 391IANaS001
`~ a”~nwL
`SITSAOIAgG
`if IT
`L SldcolASIAAd LNAIT9
`
`|
`
`|
`
`‘ YYOMLAN A _FOIAAd <>
`. GaNI3a | YSOMLAN\ SYVMLAOS +
`HATIONANOD < [ponrusimay50T SOF
`
`U.S. Patent
`
`Feb. 16, 2016
`
`Sheet 1 of 9
`
`US 9,264,301 B1
`
`aa
`
`

`

`ALIMNDS SOILSILVLS INSAS sa1nu 714qd0W NasBEL SET VEL aTLINN 1OU¥LNOD LINN LINN un
`SPL anT vrL sor —_
`NOLLVAYOSNI qyoo3Hy
`OFT cri orl
`—_ OSTcSt
`ASIANAS HLVWd H.LWd ONITACOW NaS
`ASIAaG NOILVOISIMSA NOILLVNIWYA Lad
`Vv A
`AOVAMALNI NGS wasn
`A A
`
`U.S. Patent
`
`YATIOUNLNOD NGS
`
`ch
`
`LINN TONLNOD
`
`oer
`
`Feb. 16, 2016
`
`Sheet 2 of 9
`
`US 9,264,301 B1
`
`A9VANALNI
`
`é Old
`
`

`

`0¢z2 |NOILVLSYYOM
`YATIONLNOD LNas B02
`NIWGV 90ZTIVMaUl4
`MaAaS gam Sal AXOUd SSM
`o1z B Pile ale 5
`gzz—| 92 pzz— zzz—|
`
`YaLnowd 202
`
`SIN3I1D €aM
`
`€ Slsa
`
`U.S. Patent
`
`Feb. 16, 2016
`
`Sheet 3 of 9
`
`US 9,264,301 B1
`
`8rz
`
`HOLIMS
`
`Ore
`
`A
`
`

`

`AYOLOANIG 39VAYSLNI YaSNAAILOV997 l
`—_ SNIONA 967tle NOILVOLLNSHLNV MaAuas
`LINN IMd IN3W39VNVIN ySeMO14 SNOILVOMIddV
`__ OZ¢ YOLINOW WALSAS
`dvq1 302 Nas aANISNA YOLIGS ADI10d
`— AdMNOd —
`ple Baz
`97z YATIONLNOD
`YaAusaS SHHOLIMS 4<—_—__ >
`B0z AT ISVOILLINW
`p9t TMSaOVNVAJOYNOSaY aT 4ANV 1d TOYLNOD
`HHOMLAN LN3I19 dol
`— 92
`
`987<—$_> aounos
`ANV1d VLVd
`
`Feb. 16, 2016
`
`Sheet 4 of 9
`
`US 9,264,301 B1
`
`U.S. Patent
`
`08z
`
`Pv Sls
`
`

`

`U.S. Patent
`
`Feb. 16, 2016 Sheet 5 of 9
`
`US 9,264,301 B1
`
`DETERMINE CONNECTIONS
`
`BETWEEN NETWORK DEVICES
`
`f
`
`DETERMINE SERVICE DEVICES
`
`OF NETWORK DEVICES
`
`300
`YY
`
`302
`“a
`
`DETERMINE ZONES FOR
`
`304
`
`PACKET FLOWS THROUGH
`
`NETWORK DEVICES
`¥
`
`DETERMINE TRUSTED
`
`PACKET FLOWS
`
`306/
`
`|
`
`DETERMINE PATHS BASED ON
`
`CONNECTIONS, SERVICES,
`
`ZONES, AND/OR TRUSTED
`
`PACKET FLOWS
`
`308
`
`{
`
`PROGRAM NETWORK DEVICES
`
`TO FORWARD TRAFFIC ALONG
`
`DETERMINED PATHS
`
`|
`
`MONITOR NETWORK DEVICES
`
`!
`
`REPROGRAM NETWORK
`
`DEVICES IN RESPONSE TO
`
`IMPROPER OPERATION
`
`FIG. 5
`
`

`

`U.S. Patent
`
`Feb. 16, 2016 Sheet 6 of 9
`
`US 9,264,301 B1
`
`PROGRAM NETWORK DEVICES
`
`330Y
`
`TO SEND PACKETS TO
`
`SERVICE DEVICE
`y
`PROGRAM NETWORK DEVICES
`
`TO RECEIVE PACKETS FROM
`
`SERVICE DEVICE
`v
`CONFIGURE SERVICE DEVICE
`
`TO SEND SERVICE-RELATED
`
`DATA TO NETWORK DEVICES
`
`:
`
`PROGRAM NETWORK DEVICES
`
`TO PERFORM PROGRAMMED
`
`ACTION BASED ON SERVICE-
`
`RELATED DATA
`
`!
`
`RECEIVE REPORTING DATA
`
`FROM NETWORK DEVICES FOR
`
`PROGRAMMED ACTION
`
`332
`f
`
`334
`
`yy
`
`yr
`
`FIG. 6
`
`

`

`U.S. Patent Feb. 16, 2016 Sheet 7 of 9 US 9,264,301 B1
`
`PROGRAM NETWORK DEVICES 990
`
`TO UTILIZE PATH FORA
`
`PACKET FLOW
`
`f
`352
`SEND TEST PACKETS ALONG /~
`
`PACKET FLOW
`
`f
`
`DETERMINE WHETHER TEST
`
`PACKETS SENT CORRECTLY
`
`v
`
`PRESENT REPORT BASED ON
`
`DETERMINATION
`
`354
`
`y3e
`
`FIG. 7
`
`

`

`U.S. Patent
`
`Feb. 16, 2016 Sheet 8 of 9 US 9,264,301 Bl
`
`370
`PROGRAM PRIMARY PATHS f
`FOR NETWORK DEVICES
`
`t
`PROGRAM BACKUP PATHS TO yore
`PRIMARY PATHS FOR
`
`NETWORK DEVICES
`
`——> u st
`
`> MONITOR NETWORK DEVICES
`
`: a
`No / FAILURE OF NETWORK
`DEVICE ALONG PRIMARY
`
`\ PATH?
`
`YES |
`
`CONFIGURE NETWORK 378
`DEVICES TOUSEBACKUP |
`PATH FOR FAILED
`
`PRIMARY PATH
`
`FIG. 8
`
`

`

`U.S. Patent
`
`Feb. 16, 2016 Sheet 9 of 9
`
`US 9,264,301 B1
`
`BEGIN AUTHENTICATION
`
`SESSION WITH CLIENT DEVICE
`
`{
`
`RECEIVE CREDENTIALS
`
`FROM CLIENT DEIVCE
`
`t
`
`TERMINATE
`
`AUTHENTICATION SESSION
`
`y
`
`DETERMINE LOCATION OF
`
`CLIENT DEVICE
`
`390
`YY
`
`392
`f
`
`394
`
`396/
`
`!
`
`DETERMINE APPLICABLE
`
`/-398
`
`POLICIES FOR CLIENT DEVICE
`
`!
`
`PROGRAM NETWORK DEVICES
`
`TO ENFORCE POLICIES
`
`400
`
`!
`
`PERFORM DPI ON PACKET(S)
`
`FROM CLIENT DEVICE
`
`402
`
`FIG. 9
`
`

`

`US 9,264,301 Bl
`
`1
`
`2
`
`HIGH AVAILABILITY FOR SOFTWARE
`
`connections between the network devices, determine one or
`
`DEFINED NETWORKS
`
`more paths for network traffic between the network devices
`
`based on the determination of the connections, and program
`
`This application claims the benefit of U.S. Provisional
`
`the network devices to direct network traffic along the one or
`
`Application No. 61/703,692, filed Sep. 20, 2012, the entire
`
`more paths.
`
`contents of which are hereby incorporated by reference.
`
`TECHNICAL FIELD
`
`This disclosure relates to computer networks and, in par-
`
`ticular, software defined networks.
`
`BACKGROUND
`
`In another example, a computer-readable storage medium
`
`has stored thereon instructions that, when executed, cause a
`
`processor of a controller device for a software defined net-
`
`work to determine connections between network devices in
`
`the software defined network, determine one or more paths
`
`for network traffic between the network devices based on the
`
`determination of the connections, and program the network
`
`devices to direct network traffic along the one or more paths.
`
`In another example, a method includes programming, by a
`
`Computer networks are formed by a collection of devices
`
`controller device for a software defined network (SDN), a first
`
`operating according to a set of various protocols. Typical
`
`network device of the SDN to send packets ofa packet flow to
`
`computer networks are formed by routers that each executes
`
`a service device, and programming, by the controller device,
`
`routing protocols to discover routes between various end-
`
`one or more network devices of the SDN to perform a pro-
`
`points, and switches that execute switching protocols, such as
`
`grammed action on packets of the packet flow based on data
`
`the spanning tree protocol (STP). In accordance with the
`
`20
`
`received from the service device for the packet flow.
`
`routing protocols, routers exchange information regarding
`
`In another example, a controller device for a software
`
`routes discovered by the routers. In this manner, each router
`
`defined network (SDN) includes one or more network inter-
`
`typically performs its own determination as to the best routes
`
`faces configured to communicate with network devices of the
`
`to use to cause traffic to traverse the network.
`
`SDN, and one or more processors configured to program a
`
`Recently, software defined networks have been developed
`
`25
`
`first network device of the SDN to send packets of a packet
`
`as a way to deploy and operate networks and develop new
`
`flow to a service device, and program one or more network
`
`networking applications. In general, software defined net-
`
`devices of the SDN to perform a programmed action on
`
`works involve the use ofa standalone controller that performs
`
`packets of the packet flow based on data received from the
`
`the control functionality for a set of network devices. As an
`
`service device for the packet flow.
`
`example of software defined networking, in the case of rout-
`
`30
`
`In another example, a computer-readable storage medium
`
`ing, rather than routers performing individual analyses to
`
`has stored thereon instructions that, when executed, cause a
`
`determine routes through the network, the controller can
`
`processor of a controller device for a software defined net-
`
`determine the routes and program other devices in the net-
`
`work (SDN) to program a first network device of the SDN to
`
`work to behave according to the determinations made by the
`
`send packets of a packet flow to a service device, and program
`
`35
`
`controller. Different protocols may be used to implement
`
`one or more network devices of the SDN to perform a pro-
`
`software defined networking, including open protocols like
`
`grammed action on packets of the packet flow based on data
`
`OpenFlow, and proprietary protocols from network vendors.
`
`received from the service device for the packet flow.
`
`OpenFlow is described in McKeown et al., “OpenFlow:
`
`In another example, a method includes programming, by a
`
`Enabling Innovation in Campus Networks,’ http://ww-
`
`controller device for a software defined network (SDN), a set
`
`w.openflow.org//documents/openflow-wp-latest.pdf, which
`
`40
`
`of network devices of the SDN to form a path through the
`
`is hereby incorporated by reference in its entirety.
`
`SDN and to send data representative of packets sent along the
`
`path to the controller device, sending, by the controller
`
`SUMMARY
`
`device, packets of a packet flow corresponding to the path to
`
`one of the set of network devices, determining, by the con-
`
`In general, this disclosure describes techniques related to
`
`45
`
`troller device, whether the set of network devices is properly
`
`controlling software defined networks (SDNs). A software
`
`forwarding the packets of the packet flow along the path based
`
`defined network is generally a network of interconnected
`
`on data received from the set of network devices, and present-
`
`computing devices having forwarding planes or data planes
`
`ing a report representative of the determination.
`
`that can be programmed remotely by one or more controller
`
`In another example, a controller device for a software
`
`devices. In this manner, the control plane can be physically
`
`30
`
`defined network (SDN) includes one or more network inter-
`
`separate from the data plane (or forwarding plane) for an
`
`faces configured to communicate with network devices of the
`
`SDN. These computing devices can have either physical
`
`SDN, and one or more processors configured to program a set
`
`instantiation or virtual (software-only) instantiation without
`
`of network devices of the SDN to form a path through the
`
`the presence of a hardware appliance. This disclosure
`
`SDN and to send data representative of packets sent along the
`
`describes various techniques related to controlling SDNs.
`
`55
`
`path to the controller device, send, via one of the network
`
`Tn one example, a method includes determining, by a con-
`
`interfaces, packets of a packet flow corresponding to the path
`
`troller device for a software defined network, connections
`
`to one of the set of network devices, determine whether the set
`
`between network devices in the software defined network,
`
`of network devices 1s properly forwarding the packets of the
`
`determining, by the controller device, one or more paths for
`
`packet flow along the path based on data received from the set
`
`network traffic between the network devices based on the
`
`60
`
`of network devices, and present a report representative of the
`
`determination of the connections, and programming, by the
`
`determination.
`
`controller device, the network devices to direct network traf-
`
`In another example, a computer-readable storage medium
`
`fic along the one or more paths.
`
`has stored thereon instructions that, when executed, cause a
`
`In another example, a controller device for a software
`
`processor of a controller device for a software defined net-
`
`defined network includes one or more interfaces for commu-
`
`65
`
`work (SDN) to program a set of network devices of the SDN
`
`nicating with network devices in the software defined net-
`
`to form a path through the SDN and to send data representa-
`
`work, and one or more processors configured to determine
`
`tive of packets sent along the path to the controller device,
`
`

`

`US 9,264,301 Bl
`
`3
`
`4
`
`send packets of a packet flow corresponding to the path to one
`
`wherein the software-based switch description objects each
`
`of the set of network devices, determine whether the set of
`
`comprise data representative of a respective primary control-
`
`network devices is properly forwarding the packets of the
`
`ler corresponding to a controller of the one or more control-
`
`packet flow along the path based on data received from the set
`
`lers, a respective switch identifier, a respective switch media
`
`of network devices, and present a report representative of the
`
`determination.
`
`In another example, a method includes determining, by a
`
`controller device for a software defined network (SDN), a
`
`primary path for network traffic between network devices of
`
`the SDN, determining, by the controller device, a backup path
`
`to the primary path for the network traffic between the net-
`
`work devices of the SDN, and, in response to a failover event,
`
`configuring the network devices to switch from the primary
`
`path to the backup path.
`
`In another example, a controller device for a software
`
`defined network (SDN) includes one or more network inter-
`
`faces configured to communicate with network devices of the
`
`SDN, and one or more processors configured to determine a
`
`access control (MAC) address, a respective Internet protocol
`
`(IP) address, and a respective array of port description objects
`
`for each physical port on the respective switch, and manag-
`
`ing, Via the network interfaces, at least some of the switches
`
`using the switch description objects and the controller
`
`objects.
`
`In another example, a computer-readable storage medium
`
`has stored thereon instructions that, when executed, cause a
`
`processor of a controller device for a software defined net-
`
`work (SDN) to obtain data representative of physical network
`
`devices in the SDN, wherein the physical network devices
`
`comprise one or more switches and one or more controllers
`
`including the controller device, instantiate software-based
`
`primary path for network traffic between network devices of
`
`controller objects for each of the one or more controllers,
`
`the SDN, determine a backup path to the primary path for the
`
`20
`
`wherein the software-based controller objects each comprise
`
`network traffic between the network devices of the SDN, and,
`
`data representative of a respective instance name, a respective
`
`in response to a failover event, configure the network devices
`
`server hostname, and a respective server host port, instantiate
`
`to switch from the primary path to the backup path.
`
`software-based switch description objects for each of the one
`
`In another example, a computer-readable storage medium
`
`or more switches, wherein the software-based switch descrip-
`
`has stored thereon instructions that, when executed, cause a
`
`25
`
`tion objects each comprise data representative of a respective
`
`processor of a controller device for a software defined net-
`
`work (SDN) to determine a primary path for network traffic
`
`between network devices of the SDN, determine a backup
`
`path to the primary path for the network traffic between the
`
`network devices of the SDN, and, in response to a failover
`
`30
`
`event, configure the network devices to switch from the pri-
`
`mary path to the backup path.
`
`In another example, a method includes obtaining, by a
`
`controller device of a software defined network (SDN), data
`
`primary controller corresponding to a controller of the one or
`
`more controllers, a respective switch identifier, a respective
`
`switch media access control (MAC) address, a respective
`
`Internet protocol (IP) address, and a respective array of port
`
`description objects for each physical port on the respective
`
`switch, and manage at least some of the switches using the
`
`switch description objects and the controller objects.
`
`In another example, a method includes receiving, by a
`
`representative of physical network devices in the SDN,
`
`35
`
`wherein the physical network devices comprise one or more
`
`switches and one or more controllers including the controller
`
`device, instantiating, by the controller device, software-based
`
`controller objects for each of the one or more controllers,
`
`controller device for a software defined network (SDN), cre-
`
`dentials from a client device in accordance with a public key
`
`infrastructure (PKI)-based authentication protocol, deter-
`
`mining, by the controller device, one or more policies that are
`
`applicable to the client device based on the received creden-
`
`wherein the software-based controller objects each comprise
`
`40
`
`tials, and programming, by the controller device, network
`
`data representative of a respective instance name, a respective
`
`devices of the SDN to enforce the determined policies on a
`
`server hostname, and a respective server host port, instanti-
`
`per-packet-flow basis for packet flows including the client
`
`ating, by the controller device, software-based switch
`
`device.
`
`description objects for each of the one or more switches,
`
`In another example, a controller device for a software
`
`wherein the software-based switch description objects each
`
`45
`
`defined network (SDN) includes one or more network inter-
`
`comprise data representative of a respective primary control-
`
`faces configured to communicate with network devices of the
`
`ler corresponding to a controller of the one or more control-
`
`lers, a respective switch identifier, a respective switch media
`
`access control (MAC) address, a respective Internet protocol
`
`(IP) address, and a respective array of port description objects
`
`30
`
`for each physical port on the respective switch, and manag-
`
`ing, by the controller device, at least some of the switches
`
`using the switch description objects and the controller
`
`objects.
`
`In another example, a controller device for a software
`
`55
`
`defined network (SDN) includes one or more network inter-
`
`faces configured to communicate with network devices of the
`
`SDN, and one or more processors configured to obtain data
`
`representative of physical network devices in the SDN,
`
`wherein the physical network devices comprise one or more
`
`60
`
`SDN, and one or more processors configured to receive cre-
`
`dentials from a client device in accordance with a public key
`
`infrastructure (PKI)-based authentication protocol, deter-
`
`mine one or more policies that are applicable to the client
`
`device based on the received credentials, and program net-
`
`work devices of the SDN to enforce the determined policies
`
`on a per-packet-flow basis for packet flows including the
`
`client device.
`
`In another example, a computer-readable storage medium
`
`has stored thereon instructions that, when executed, cause a
`
`processor of a controller device for a software defined net-
`
`work (SDN) to receive credentials from a client device in
`
`accordance with a public key infrastructure (PKI)-based
`
`switches and one or more controllers including the controller
`
`authentication protocol, determine one or more policies that
`
`device, instantiate software-based controller objects for each
`
`are applicable to the client device based on the received
`
`of the one or more controllers, wherein the software-based
`
`credentials, and program network devices of the SDN to
`
`controller objects each comprise data representative of a
`
`enforce the determined policies on a per-packet-flow basis for
`
`respective instance name, a respective server hostname, and a
`
`65
`
`packet flows including the client device.
`
`respective server host port, instantiate software-based switch
`
`The details of one or more examples are set forth in the
`
`description objects for each of the one or more switches,
`
`accompanying drawings and the description below. Other
`
`

`

`US 9,264,301 Bl
`
`5
`
`6
`
`features, objects, and advantages will be apparent from the
`
`ally or alternatively include malware detection devices, net-
`
`description and drawings, and from the claims.
`
`work anti-virus devices, network packet capture and analysis
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`devices, domain name service (DNS) and global DNS server
`
`devices, honeypot devices, reflector net devices, tar pit
`
`FIG. 1 is a block diagram illustrating an example system in
`
`Service devices 116 may, additionally or alternatively,
`
`devices, mail proxies, and anti-spam devices.
`
`which various techniques of this disclosure may be used.
`
`FIG. 2 is a block diagram illustrating an example set of
`
`components of a software defined network (SDN) controller.
`
`FIG. 3 is a conceptual diagram illustrating an example
`
`system including various devices that may be used in accor-
`
`dance with the techniques of this disclosure.
`
`FIG. 41s aconceptual diagram illustrating another example
`
`system including various devices that may be used in accor-
`
`dance with the techniques of this disclosure.
`
`FIG. 5 is a flowchart illustrating an example method for
`
`constructing paths in an SDN.
`
`FIG. 6 is a flowchart illustrating an example method for
`
`directing network traffic of an SDN to a service device.
`
`include devices in various device categories such as, for
`
`example, network and application security devices, applica-
`
`tion optimization devices, scaling devices, traffic shaping
`
`devices, and/or monitoring and analytics devices. Moreover,
`
`although shown as individual devices, it should be understood
`
`that service devices may be realized by physical devices,
`
`multi-tenant devices, or using virtual services (e.g., cloud-
`
`based services). Moreover, service devices 116 may represent
`
`multi-function devices. For purposes of example and ease of
`
`explanation, this disclosure primarily describes individual
`
`service devices. However, it should be understood that the
`
`techniques of this disclosure may be readily applied to virtual
`
`FIG. 7 is a flowchart illustrating an example method for
`
`20
`
`devices and cloud-based applications, in addition or in the
`
`using test traffic to determine whether an SDN packet flow is
`
`alternative to physical devices. Likewise, where this disclo-
`
`operating correctly.
`
`sure refers to a switch or other network device, it should be
`
`FIG. 8 is a flowchart illustrating an example method for
`
`understood that these techniques may apply to virtual
`
`using one or more backup paths to a primary path through an
`
`switches or other virtual network devices.
`
`SDN.
`
`25
`
`SDN controller 112 may implement any or all of the tech-
`
`FIG. 9 is a flowchart illustrating an example method for
`
`niques described in this disclosure for controlling various
`
`performing authentication and authorization techniques
`
`devices of SDN 106, such as network devices 108, 110, and
`
`described herein.
`
`service devices 116. These techniques are described in
`
`greater detail below. Examples of the techniques described
`
`DETAILED DESCRIPTION
`
`30
`
`below include a data model for managing devices ofan SDN,
`
`implementing high availability among devices of an SDN,
`
`FIG. 1 is a block diagram illustrating an example system
`
`performing services for packets flowing along a path through
`
`100 in which various techniques of this disclosure may be
`
`an SDN, trigger-driven policy distribution, and using test
`
`used. In this example, system 100 includes software defined
`
`traffic to verify whether devices of the SDN are operating
`
`35
`
`correctly.
`
`network (SDN) 106, which includes network devices 108,
`
`110 and service devices 116. Network devices 108, 110 may
`
`Users of SDN controller 112 may be provided with the
`
`comprise switches, and other devices (not shown). These
`
`ability to create programs that are executable by SDN con-
`
`network devices can be physical instantiations or virtual
`
`troller 112. For example, a software development kit (SDK)
`
`instantiations. SDN 106 may also include other types of
`
`may be provided for creating programs that can be executed
`
`devices, such as routers, load balancers, various L4-L7 net-
`
`40
`
`by SDN controller 112. In this manner, users or other third
`
`work devices, or even multi-tenant capable devices, among
`
`parties may utilize an environment to write applications that
`
`other network devices. Again, these network devices can be
`
`can be executed by SDN controller 112. For example, SDN
`
`either of physical or virtual instantiations. In general, network
`
`controller 112 itself may host one or more L4-L7 services,
`
`devices of SDN 106, such as network devices 108,110, can be
`
`rather than (or in addition to) services provided by service
`
`programmed to forward network traffic. System 100 also
`
`45
`
`devices 116. As another example, a L3 application (such as
`
`includes software defined network (SDN) controller 112.
`
`Quagga, a Layer 3 routing stack) may be executed by SDN
`
`Administrator 114 uses SDN controller 112 to control (thatis,
`
`controller 112. In this case, the Quagga routing stack would
`
`program) network devices of SDN 106, such as network
`
`be running a standard routing protocol such as OSPF (Open
`
`devices 108, 110.
`
`Shortest Path Forwarding) or BGP (Border Gateway Proto-
`
`SDN 106 generally serves to interconnect various endpoint
`
`30
`
`col) and would update its routing tables like before, except
`
`devices, such as client device 102 and server device 104. In
`
`that it would use the SDN controller 112 to command the
`
`addition, SDN 106 may provide services to network traffic
`
`network equipment to update their forwarding tables.
`
`flowing between client device 102 and server device 104.
`
`SDN controller 112 may be configured to logically repre-
`
`Alternatively, SDN 106 may provide services to client device
`
`sent an existing physical or virtual topology within a de-
`
`102, without further directing traffic to server device 106. For
`
`55
`
`militarized zone (DMZ) in an organizational network, such
`
`example, administrator 114 may use SDN controller 112 to
`
`that the logical topology can be manipulated with corre-
`
`program network devices of SDN 106 to direct network traffic
`
`sponding physical impact on flows. SDN controller 112 may
`
`for client device 102 to one or more of service devices 116.
`
`also be used in combination with a switching platform into
`
`Service devices 116 may include, for example, intrusion
`
`which existing physical devices are plugged, and a software
`
`detection service (IDS) devices, intrusion prevention system
`
`60
`
`control platform that is able to virtually map and control the
`
`(IPS) devices, web proxies, web servers, web-application
`
`logical flows running through the switch. The switching plat-
`
`firewalls and the like. In other examples, service devices 116
`
`form may be powered by ASICs (application specific inte-
`
`may, additionally or alternatively, include devices for provid-
`
`grated circuits) and provide flow control capabilities as
`
`ing services such as, for example, denial of service (DoS)
`
`exposed by proprietary vendor-specific APIs or open stan-
`
`protection, distributed denial of service (DDoS) protection,
`
`65
`
`dards such as OpenFlow. Alternatively, the switching plat-
`
`traffic filtering, wide area network (WAN) acceleration, or
`
`form may be based on software, while providing requisite
`
`other such services. Service devices 116 may also addition-
`
`hardware for executing the software (e.g., computer-readable
`
`

`

`US 9,264,301 Bl
`
`7
`
`8
`
`media for storing instructions and one or more processing
`
`like. The paths may be determined on a per-packet basis or on
`
`units for executing the instructions).
`
`a per-packet-flow basis, in various examples.
`
`Devices that may be plugged into (that is, communicatively
`
`SDN controller 112 may also be configured to track
`
`coupled to) SDN controller 112 (also sometimes referred to
`
`changes related to rules provided for modifying paths through
`
`as a “FlowDirector’) generally include classes of devices
`
`SDN 106. As an example, SDN controller 112 may track
`
`found in most network-based DM/7Zs, including firewalls, web
`
`identifying information for a user who added, modified, or
`
`proxies, mail proxies, AV (anti-virus) proxies, mail systems,
`
`removed a rule. SDN controller 112 may store, track, and
`
`IDS (ntrusion detection systems), IPS (intrusion prevention
`
`report such information for any or all rules used by SDN
`
`systems), VPN (virtual private network) servers, web appli-
`
`controller 112. In this manner, for any or all rules of SDN
`
`cation firewalls, vulnerability scanners, network recording
`
`controller 112, information may be available regarding who
`
`and analysis systems, and packet shapers. Most of these
`
`made the rule, where the rule was made as part of Rule Flows,
`
`devices are either security devices, or traffic engineering or
`
`how SDN controller 112 received the rule (e.g., via an appli-
`
`visibility devices, in some examples.
`
`cation programming interface (API), a graphical user inter-
`
`SDN controller 112 may overcome certain problems
`
`