(12) United States Patent
`(10) Patent N0.:
`US 6,779,016 B1
`Aziz et al.
`(45) Date of Patent:
`*Aug. 17, 2004
`
`U8006779016B1
`
`(54) EXTENSIBLE COMPUTING SYSTEM
`
`(75)
`
`Inventors: Ashar Aziz, Fremont, CA (US); T0111
`Markson, San Mateo, CA (US);
`Martin Patterson, Mountain View, CA
`(US)
`
`(73) Assignee: Terraspring, Inc., Fremont, CA (US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`W0
`W0
`WO
`
`12/1997
`WO 97/49214 A1
`11/1999
`WO 99/57957 A2
`5/2000
`\VO 00/29954 A1
`OTHER PUBLICATIONS
`
`Eric l-‘rud’hommeaux, “XML—based HTTP Server Configu-
`ration Language,” http://'www.w3.org/l999/07/97httpiserv—
`er—confhtml, Sep. 7, 1999, XP—002152599, pp. 1—5.
`
`(List continued on next page.)
`Primary anmz'ner—Nabil El-I-Iady
`(74) Attorney, Agent, or Firm—Hickman Palermo Truong
`& Becker LLP; Edward A. Becker
`
`This patent is subject to a terminal dis—
`claimer.
`
`(5 7)
`
`ABSTRACT
`
`(21) Appl. N0.: 09/502,170
`
`(22)
`
`Filed:
`
`Feb. 11, 2000
`
`Related US. Application Data
`Provisional application No. 60/150,394, filed on Aug. 23,
`1999.
`
`(60)
`
`Int. Cl.7 ................................................ G061“ 15/16
`(51)
`(52) US. Cl.
`....................... 709/201; 709/104; 709/105;
`709/107; 709/205; 709/208; 709/209; 709/226
`(58) Field Of Search ................................. 709/201, 202,
`709/205, 226, 107, 104, 105, 208, 209
`
`(56)
`
`References Cited
`U.S, PATENT DOCUMENTS
`
`4,591,967 A
`5,163,130 A
`5,504,670 A
`
`5/1986 Mattes et a1.
`11/1992 Hullot
`4/1996 Barth et a1.
`
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`
`0262750 A2
`0 262 750 A2
`0 490 624 A2
`0 750 256 A2
`0 791 881 A1
`0 905 621 Al
`0917056 A2
`0935200 A1
`
`4/1988
`4/1988
`6/1992
`12/1996
`8/1997
`3/1999
`5/1999
`8/1999
`
`Methods and apparatus providing a dynamically sized,
`highly scalable and available server farm are disclosed. A
`Virtual Server Farm (VSF) is created out of a wide scale
`computing fabric (“Computing Grid”) which is physically
`constructed once and then logically divided up into VSFs for
`various organizations on demand. Each organization retains
`independent administrative control of a VSF. A VSF is
`dynamically firewalled within the Computing Grid. A allo-
`cation and control of the elements in the VSF is performed
`by a Control Plane connected to all computing, networking,
`and storage elements in the computing grid through soecial
`control ports. The internal topology of each VSF is under
`control of the Control Plane. N0 physical rewiring is nec-
`essary in order to construct VSFs in many di erent
`configurations, including single-tier Web server or multi-tier
`Web-server, application server, database server con lgura-
`tions. Each tier of a multi—ticr VSF (c.g. Web server tier,
`application server tier, database server tier, etc) can be
`dynamically sized based on the load on the servers in that
`particular tier. Storage devices may include a plurality of
`pre—delined logical blueprints that are associated with roles
`that may be assumed by the computing grid elements.
`Initially, no computing element is dedicated to any particular
`role or task such as Web server, application server, database
`server, etc. The role of each computing element is acquired
`from one of a plurality of pre—deflned, stored blueprints,
`each of which defines a boot
`image for the computing
`elements that are associated with that role.
`
`
`
`
`49 Claims, 17 Drawing Sheets
`
`200
`
`DISK1
`DISK2
`
`CONTROL
`PLANE
`mg
`SAN CONTROL
`
`
`
`
`CPU CONTROL
`208
`
`
`
`
`
`VLAN CONTROL
`
`
`
`
`INTERNET
`
`Oracle Exhibit 1006, Page 1
`
`Oracle Exhibit 1006, Page 1
`
`

`

`us 6,779,016 B1
`
`Page 2
`
`US PATENT DOCUMENTS
`
`OTIIER PUBLICATIONS
`
`.............. 395/653
`
`112/832 Emmi? Ct al'
`29:33:33: 2
`Jose 0
`. ,.
`,
`1
`8/1997 George ct al.
`5,659,786 A
`5/1998 Raab ct 211.
`5,751,967 A
`10/1998 Hansen
`5,819,042 A
`,
`538213937 A ,, 10/1998 Tonellietal.
`5,878,232 A *
`3/1999 Marirnuthu ................. 709/249
`5,951,683 A
`9/1999 Yuuki et a1.
`5,974,563 A
`10/1999 Beeler, Jr.
`6,182,139 B1 "
`1/2001 Brendel
`...................... 709/226
`
`6,219,699 B1 "
`4/2001 McCloghrie et a].
`. 709/221
`................ 709/221
`6,223,218 B1 "
`4/2001 Iijima et al.
`6,230,200 B1
`5/2001 Forecast et al.
`6,381,321 B1 2
`4/2002 Brown el al.
`llllllllll 379,207.02
`6,389,432 131 1
`5/2002 Pothapragada et al.
`..... 707/205
`6,446,141 B1
`9/2002 Nolan el al.
`6,466,559 B1 X 10/2002 Johansson et al.
`.......... 370/335
`. 709/107
`6,505,229 B1 X
`1/2003 Tur11e1 et al.
`.
`
`..................... 700/3
`6,597,956 B1 *
`7/2003 Aziz et al.
`
`Nathan J. Muller, “Design and Conquer,” Oct. 1996, BYTE,
`7
`VOL 21’ NO' 10’ XP.000683573’7PP' 93 98'
`~
`~
`.
`Rob Short, et al., “Windows NT Clusters for Availability and
`Scalability,” 1997 IEEE, pp. 8—13.
`Elliotte Rusty IIarold, “XML: Extensible Markup Lan-
`ma ‘6 77 1998
`1_431
`5
`5 ’
`’PP'
`'
`Radek Vingralck, et a1., “Snowball: Scalable Storage on
`Networks of Workstations With Balanced Load,” pp.
`1177156, Distributed and Parallel Databases, V01, 6, N0. 2,
`Apr. 1998, XP—002162201.
`7
`7
`“
`Armando’Fox, et a1., ClusteriBased Scalable NetVsork
`SeerCCS>
`PP~
`78—91, Operatmg
`SYStemS
`ReVlCW
`(SIGOPS), US, ACM Headquarter, New York, vol. 31, No.
`5, DEC. 1, 1997, XP—000771023.
`
`* cited by examiner
`
`Oracle Exhibit 1006, Page 2
`
`Oracle Exhibit 1006, Page 2
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 1 0f 17
`
`US 6,779,016 B1
`
`SINGLE MACHINE
`
`DISK
`
`104
`
`CPU
`
`102
`
`105 HNTERNET
`
`979. 0-6
`
`PRIOR ART
`
`LOAD BALANCER/FIREWALL
`
`112
`
`110
`
`106HNTERNET
`
`979. 7%
`
`PRIOR ART
`
`Oracle Exhibit 1006, Page 3
`
`Oracle Exhibit 1006, Page 3
`
`

`

`US. Patent
`
`27,1«.5uA
`
`m3
`
`4
`
`mS
`
`2
`
`7
`
`SU
`
`09,
`
`1B
`
`a.
`
`mj<>>wm_u_2mozs<m28
`
`525252
`
`mE<moan
`
`7\§7.6,awkm,
`
`wm<m§<o
`
`mmm>mmm
`
`wmm>mmm
`
`zo_._.<o_._n_n_<
`
`Oracle Exhibit 1006, Page 4
`
`Oracle Exhibit 1006, Page 4
`
`
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 3 0f 17
`
`US 6,779,016 B1
`
`AOmHzoo
`
`mz<4m
`
`
`
`ammmzotgmz<m
`
` gm
`
`2..meNme
`
`SEE
`
`
`
`405onz<w
`
`
`
`Joy—H200:ao
`
`
`
`
`
`mol._-2.50loo—N
`
`
`
`405200513
`
`
`
`Oracle Exhibit 1006, Page 5
`
`Oracle Exhibit 1006, Page 5
`
`
`
`
`
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 4 0f 17
`
`US 6,779,016 B1
`
`306b
`
`
`
`3063
`
`304
`
`SAN ZONE X
`
`-
`
`
`
`
`
`LB/FIREWALL
`
`106‘1NTERNET
`
`99.3
`
`Oracle Exhibit 1006, Page 6
`
`Oracle Exhibit 1006, Page 6
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 5 0f 17
`
`US 6,779,016 B1
`
`400
`
`IDLE POOL
`
`VSF1
`
`VSF2
`
`Oracle Exhibit 1006, Page 7
`
`Oracle Exhibit 1006, Page 7
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 6 0f 17
`
`US 6,779,016 B1
`
`
`CECE-4
`
`402
`
`IDLE POOL
`
`
`
`VSF1
`
`VSFZ
`
`Oracle Exhibit 1006, Page 8
`
`Oracle Exhibit 1006, Page 8
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 7 0f 17
`
`US 6,779,016 B1
`
`400
`
`
`SEES-~-
`IDLE POOL
`
`
`
`VSF1
`
`VSF2
`
`Oracle Exhibit 1006, Page 9
`
`Oracle Exhibit 1006, Page 9
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 8 0f 17
`
`US 6,779,016 B1
`
`
` 400
`CECE-~-
`
`IDLE POOL
`
`VSF1
`
`VSF2
`
`Oracle Exhibit 1006, Page 10
`
`Oracle Exhibit 1006, Page 10
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 9 0f 17
`
`US 6,779,016 B1
`
`mom
`
`405.200
`
`c—m
`
`Iota/mz<w
`
`Ezmwhz
`
`:2523>;
`
`:.>S>
`
`m>m>
`
`Sm>m>w)m>N>
`
`
`
`gflmv§n§u2:
`
`Oracle Exhibit 1006, Page 11
`
`Oracle Exhibit 1006, Page 11
`
`
`
`
`
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 10 0f 17
`
`US 6,779,016 B1
`
`95
`
`JomEoo
`
`mz<._n_
`
`._.zm_o<
`
` mz._._u<s_
`
`2m
`
`
`mv>3>m;N;:>2>m>m>
`m;Iot>>m25>S5m>m>w)m>m>
`
`mo
`
` fi<§mmE
`
`EZmWFZ
`
`o8
`
`9m
`
`mm“.%&
`
`
`
`10:25z<m
`
`Oracle Exhibit 1006, Page 12
`
`Oracle Exhibit 1006, Page 12
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 11 0f 17
`
`US 6,779,016 B1
`
`VSF 1
`
`‘06 \INTERNET
`
`9&5
`
`Oracle Exhibit 1006, Page 13
`
`Oracle Exhibit 1006, Page 13
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 12 0f 17
`
`US 6,779,016 B1
`
`VSF1
`
`106
`
`Oracle Exhibit 1006, Page 14
`
`Oracle Exhibit 1006, Page 14
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 13 0f 17
`
`US 6,779,016 B1
`
`VSF2
`
`INTESNET
`
`106
`
`97g. 6’
`
`Oracle Exhibit 1006, Page 15
`
`Oracle Exhibit 1006, Page 15
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 14 0f 17
`
`US 6,779,016 B1
`
`2 f
`
`l_l
`.—
`
`O
`
`Oracle Exhibit 1006, Page 16
`
`Oracle Exhibit 1006, Page 16
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 15 0f 17
`
`US 6,779,016 B1
`
`woor
`
`woe./mZONmw<m05
`
`
`
`wz<._n_AOEzOo
`
`NIL/I)82
`
`wooF
`
`
`
`Iwmfiz<w
`
`Nw
`
`Oracle Exhibit 1006, Page 17
`
`Oracle Exhibit 1006, Page 17
`
`
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 16 0f 17
`
`US 6,779,016 B1
`
`
`
`
`
`15.szxmo>>>mz
`
`zoozo._ mm...szmmow2%
`
`Q$64
`
`mmpzmo
`
`Oracle Exhibit 1006, Page 18
`
`Oracle Exhibit 1006, Page 18
`
`

`

`US. Patent
`
`Aug. 17, 2004
`
`Sheet 17 0f 17
`
`US 6,779,016 B1
`
`wwmw
`
`ommF
`
`gE
`
`._<oo._
`
`xmoBEz
`
`mmmfl
`
`.50:
`
`elmNI—
`
`ONE,
`
`v_z_._
`
`m.3mmnmamafiampsmz_
`
`
`”6&8_292923228$88095
`manal.”a,5158“BEE?
`
`mm>mmw
`
`flmlr
`
`
`
`MaamI:_85%E052asm$55522:Emma
`
`Q$5
`
`Oracle Exhibit 1006, Page 19
`
`Oracle Exhibit 1006, Page 19
`
`
`
`

`

`US 6,779,016 B1
`
`1
`EXTENSIBLE COMPUTING SYSTEM
`
`This application claims the benefit of Provisional Appli-
`cation No. 60/150,394, filed Aug. 23, 1999.
`FIELD OF TIIE INVENTION
`
`The present invention generally relates to data processing.
`The invention relates more specillcally to methods,
`apparatus, and mechanisms providing an extensible,
`flexible, and scalable computing system.
`BACKGROUND OF THE INVENTION
`
`Builders of Web sites and other computer systems today
`have to deal with many systems planning issues. These
`include capacity planning for normal growth, expected or
`unexpected peak demand, availability and security of the
`site, etc. Companies who wish to provide services on the
`Web have new business and service models, which are the
`areas in which they want to innovate and lead, but in order
`to do so they have to deal with the non-trivial complexity of
`designing, building and operating a large-scale Web site.
`This includes the need to grow and scale the site while it is
`operational.
`Doing all this requires linding and hiring trained person-
`nel capable of engineering and operating such a site, which
`may be potentially large and complicated. This is creating
`difficulty for many organizations, because designing, con—
`structing and operating such large sites is simply not their
`core competency.
`One response to these issues is to host an enterprise Web
`site at a third party site, co-located with other Web sites of
`other enterprises. Such outsourcing facilities are currently
`available from companies such as Exodus, AboveNet,
`GlobalCenter, etc. These facilities provide physical space,
`and redundant network and power facilities so that
`the
`enterprise customer or user need not provide them. The
`network and power facilities are shared among many enter—
`prises or customers.
`However, the users of these facilities are still required to
`do a lot of work relating to their computing infrastructure in
`the course of building, operating and growing their facilities.
`Information technology managers of the enterprises hosted
`at such facilities remain responsible for selecting, installing,
`configuring, and maintaining their own computing equip-
`ment at
`the facilities. The managers must still confront
`difficult issues such as resource planning and handling peak
`capacity.
`Even when outsourcing companies also provide comput—
`ing facilities (e.g., Digex), the facilities are no easier to scale
`and grow for the outsourcing company, because growth
`involves the same manual and error—prone administrative
`steps. In addition, problems remain with capacity planning
`for unexpected peak demand.
`Further, each Web site may have different requirements.
`For example, particular Web sites may require the ability to
`be independently administered and controlled. Others may
`require a particular type or level of security that isolates the
`Web site from all other sites that are co-located at the service
`provider. Others may require a secure connection to an
`enterprise Intranet located elsewhere.
`Also, various Web sites differ in internal topology. Some
`sites simply comprise a row of Web servers that are load
`balanced by a Web load balancer. Suitable load balancers are
`Local Director from Cisco Systems,
`Inc., BigIP from
`F5Labs, Web Director from Alteon, etc. Other sites may be
`
`u.
`
`‘10
`
`b)v.
`
`s)u.
`’
`
`40
`
`50
`
`60
`
`2
`constructed in a multi-tier fashion, whereby a row of Web
`servers handle Hypertext Transfer Protocol
`(HTTP)
`requests, but the bulk of the application logic is implemented
`in separate application servers. These application servers in
`turn may need to be connected back to a tier of database
`servers.
`
`Some of these different configuration scenarios are shown
`in FIG. 1A, FIG. 1B, and FIG. 1C. FIG. 1A is a block
`diagram of a simple Web site, comprising a single machine
`100 comprising a CPU 102 and disk 104. Machine 100 is
`coupled to the global, packet-switched data network known
`as the Internet 106, or to another network. Machine 100 may
`be housed in a co-location service of the type described
`above.
`FIG. 1B is a block diagram of a 1—tier Web server farm
`110 comprising a plurality of Web servers W'SA, WSB,
`WSC. Each of the Web servers is coupled to a load-balancer
`112 that is coupled to Internet 106. The load balancer divides
`the traflic between the servers to maintain a balanced pro-
`cessing load on each server. Load balancer 112 may also
`include or may be coupled to a firewall for protecting the
`Web servers from unauthorized traffic.
`FIG. 1C shows a 3-tier server farm 120 comprising a tier
`of Web servers W1, W2, etc., a tier of application servers A1,
`A2, etc., and a tier of database servers D1, D2, etc. The web
`servers are provided for handling IITTP requests. The appli-
`cation servers execute the bulk of the application logic. The
`database servers execute database management system
`(DBMS) software.
`Given the diversity in topology of the kinds of Web sites
`that may need to be constructed, it may appear that the only
`way for constructing large—scale Web sites is to custom build
`each one. Indeed, this is the conventional approach. Many
`organizations are separately struggling with the same issues,
`
`
`and custom building each Web site from scratch. This is
`
`
`
`ine icient and involves a significant amount of duplicate
`
`
`wor< at di ‘erent enterprises.
`S ill another problem with the conventional approach is
`
`
`resource and capacity planning. A Web site may receive
`
`
`
`vastly di ‘erent
`levels of traffic on different days or at
`
`
`different iours within each day. At peak tra ic times, the
`Web site aardware or software may be unable to respond to
`requests in a reasonable time because it is overloaded. At
`other times, the web site hardware or software may have
`excess capacity and be underutilized. In the conventional
`approach, finding a balance between having sufficient hard—
`ware and software to handle peak traffic, without incurring
`excessive costs or having over-capacity, is a difficult prob-
`lem. Many Web sites never find the right balance and
`chronically suffer from under-capacity or excess capacity.
`Yet another problem is failure induced by human error. A
`great potential hazard present in the current approach of
`using manually constructed server farms is that human error
`in configuring a new server into a live server farm can cause
`the server farm to malfunction, possibly resulting in loss of
`service to users of that Web site.
`Based on the foregoing, there is a clear need in this field
`for improved methods and apparatus for providing a com-
`puting system that is instantly and easily extensible on
`demand without requiring custom construction.
`There is also a need for a computing system that supports
`creation of multiple segregated processing nodes, each of
`which can be expanded or collapsed as needed to account for
`changes in traffic throughput. Other needs will become
`apparent in the disclosure provided in this document.
`SUMMARY OF THE INVENTION
`
`
`
`
`
`The foregoing needs and objects, and other needs and
`objects that will become apparent
`from the following
`
`Oracle Exhibit 1006, Page 20
`
`Oracle Exhibit 1006, Page 20
`
`

`

`3
`description, are achieved by the present invention, which
`comprises, in one aspect, a method and apparatus for cre—
`ating highly scalable, highly available and secure data
`processing sites, based on a wide scale computing fabric
`(“computing grid”). The computing grid is physically con-
`structed once, and then logically divided up for various
`organizations on demand. The computing grid comprises a
`large plurality of computing elements that are coupled to one
`or more VLAN switches and to one or more storage area
`network (SAN) switches. A plurality of storage devices are
`coupled to the SAN switches and may be selectively coupled
`to one or more of the computing elements through appro—
`priate switching logic and commands. One port of the
`VLAN switch is coupled to an external network, such as the
`Internet. A supervisory mechanism, layer, machine or pro-
`cess is coupled to the VLAN switches and SAN switches.
`Initially, all storage devices and computing elements are
`assigned to Idle Pools. Under program control, the supervi-
`sory mechanism dynamically configures the VLAN
`switches and SAN switches to couple their ports to one or
`more computing elements and storage devices. As a result,
`such elements and devices are logically removed from the
`Idle Pools and become part of one or more virtual server
`farms (VSFs). Each VSF computing element is pointed to or
`otherwise associated with a storage device that contains a
`boot image usable by the computing element for bootstrap
`operation and production execution.
`By physically constructing the computing grid once, and
`securely and dynamically allocating portions of the com-
`puting grid to various organizations on demand, economies ’
`of scale are achieved that are difficult to achieve when doing
`a custom build of each site.
`
`‘10
`
`b)v.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention is illustrated by way of example,
`and not by way of limitation, in the figures of the accom-
`panying drawings and in which like reference numerals refer
`to similar elements and in which:
`
`FIG. 1A is a block diagram of a simple Web site having
`a single machine topology.
`FIG. 1B is a block diagram of a one-tier Web server farm.
`FIG. 1C is a block diagram of a three-tier Web server
`farm.
`
`FIG. 2 is a block diagram of one configuration of an
`extensible computing system 200 that
`includes a local
`computing grid.
`FIG. 3 is a block diagram of an exemplary virtual server
`farm featuring a SAN Zone.
`FIG. 4A, FIG. 4B, FIG. 4C, and FIG. 4D are block
`diagrams showing successive steps involved in adding a
`computing element and removing element from a virtual
`server farm.
`
`5)u.
`
`40
`
`50
`
`FIG. 5A is a block diagram of an embodiment of a virtual
`server farm system, computing grid, and supervisory mecha—
`nism.
`
`*'
`
`FIG. 5B is a block diagram of a system in which a
`Supervisor or Control Plane server farm is protected by a
`firewall.
`
`60
`
`FIG. 6 is a block diagram of logical connections of a
`virtual server farm.
`
`FIG. 7 is a block diagram of logical connections of a
`virtual server farm.
`
`FIG. 8 is a block diagram of logical connections of a
`virtual server farm.
`
`US 6,779,016 B1
`
`4
`FIG. 9 is a block diagram of a Control Plane server farm.
`FIG. 10 is a block diagram showing connections of
`Control Plane machines to an embodiment
`that uses a
`plurality of SAN switches (“SAN mesh”).
`FIG. 11 is a block diagram of a plurality of VSFs extended
`over W'AN connections.
`
`'4.
`
`FIG. 12 is a block diagram of a computer system with
`which an embodiment may be implemented.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT
`
`A method and apparatus for providing an extensible
`computing system is described. In the following description,
`for the purposes of explanation, numerous specific details
`are set forth in order to provide a thorough understanding of
`the present invention. It will be apparent, however, to one
`skilled in the art that the present invention may be practiced
`without these specific details. In other instances, well-known
`structures and devices are shown in block diagram form in
`order to avoid unnecessarily obscuring the present inven-
`tion.
`Virtual Server Farm (VSF)
`According to one embodiment, a wide scale computing
`fabric (“computing grid”) is provided. The computing grid
`can be physically constructed once, and then logically
`divided up for various organizations on demand. A part of
`the computing grid is allocated to each of a plurality of
`enterprises or organizations. Each organization’s logical
`portion of the computing grid is referred to as a Virtual
`Server Farm (VSF). Each organization retains independent
`administrative control of its VSF. Each VSF can change
`dynamically in terms of number of CPUs, storage capacity
`and disk and network bandwidth based on real-time
`demands placed on the server farm or other factors. Each
`VSF is secure from every other organizations’ VSF, even
`though they are all logically created out of the same physical
`computing grid. AVSF can be connected back to an Intranet
`using either a private leased line or a Virtual Private Net-
`work (VPN), without exposing the Intranet to other organi—
`zations’ VSFs.
`An organization can access only the data and computing
`elements in the portion of the computing grid allocated to it,
`that is, in its VSF, even though it may exercise lull (e.g.
`super—user or root) administrative access to these computers
`and can observe all traffic on Local Area Networks (LANs)
`to which these computers are connected. This is accom—
`plished using a dynamic fire-walling scheme, where the
`security perimeter of the VSF expands and shrinks dynami-
`cally.
`Each VSF can be used to host the content and applications
`of an organization which may be accessed via the Internet,
`Intranet or Extranet.
`Configuration and control of the computing elements and
`their associated networking and storage elements is per—
`formed by a supervisory mechanism which is not directly
`accessible through any of the computing elements in the
`computing grid. For convenience,
`in this document
`the
`supervisory mechanism is called Control Plane and may
`comprise one or more processors or a network of processors.
`The supervisory mechanism may comprise a Supervisor,
`Controller, etc. Other approaches may be used, as described
`herein.
`The Control Plane runs on a completely independent set
`of computing elements assigned for supervisory purposes,
`such as one or more servers that may be interconnected in a
`network or by other means. It performs control actions on
`
`Oracle Exhibit 1006, Page 21
`
`Oracle Exhibit 1006, Page 21
`
`

`

`5
`
`‘10
`
`b)v.
`
`the computing, networking and storage elements of the
`computing grid through special control ports or interfaces of
`the networking and storage elements in the grid. The Control
`Plane provides a physical interface to switching elements of
`the system, monitors loads of computing elements in the
`system, and provides administrative and management func-
`tions using a graphical user interface or other suitable user
`interface.
`Computers running the Control Plane are logically invis-
`ible to the computers in the computing grid (and therefore in
`any specific VSF) and cannot be attacked or subverted in any
`way via elements in the computing grid or from external
`computers. Only the Control Plane has physical connections
`to the control ports on devices in the computing grid, which
`controls membership in a particular VSF. The devices in the
`computing can be configured only through these special
`control ports, and therefore computing elements in the
`computing grid are unable to change their security perimeter
`or access storage or computing devices which they are not
`authorized to do.
`Thus, a VSF allows organizations to work with computing
`facilities that appear to comprise a private server farm,
`dynamically created out of a large-scale shared computing
`infrastructure, namely the computing grid. A Control Plane
`coupled with the computing architecture described herein
`provides a private server farm whose privacy and integrity
`is protected through access control mechanisms imple—
`mented in the hardware of the devices of the computing grid.
`The internal topology of each VSF is controlled by the
`Control Plane. The Control Plane can take the basic inter— .
`connection of computers, network switches and storage
`network switches described herein and use them to create a
`variety of server farm configurations. These include but are
`not limited to, single-tier W'eb server farms front-ended by
`nu.
`a load balancer, as well as multi—tier configurations, where a .
`Web server talks to an application server, which in turn talks
`to a database server. A variety of load balancing, multi—
`tiering and fire-walling configurations are possible.
`The Computing Grid
`The computing grid may exist in a single location or may
`be distributed over a wide area. First
`this document
`describes the computing grid in the context of a single
`building-sized network, composed purely of local area tech-
`nologies. Then the document describes the case where the
`computing grid is distributed over a wide area network
`(WAN).
`FIG. 2 is a block diagram of one configuration of an
`extensible computing system 200 that
`includes a local
`computing grid 208. In this document “extensible” generally
`means that the system is flexible and scalable, having the
`capability to provide increased or decreased computing
`power to a particular enterprise or user upon demand. The
`local computing grid 208 is composed of a large number of
`computing elements CPU1, CPU2, .
`.
`. CPUn. In an exem-
`plary embodiment,
`there may be 10,000 computing .,
`elements, or more. These computing elements do not contain
`or store any long—lived per—element state information, and
`therefore may be configured without persistent or non-
`volatile storage such as a local disk. Instead, all long lived
`state information is stored separate from the computing
`elements, on disks DISKl, DISK2,
`.
`.
`. DISKn that are
`coupled to the computing elements Via a Storage Area
`Network (SAN) comprising one or more SAN Switches 202.
`Suitable SAN switches are commercially available from
`Brocade and Excel.
`All of the computing elements are interconnected to each
`other through one or more VLAN switches 204 which can
`
`40
`
`50
`
`60
`
`,—
`D
`
`US 6,779,016 B1
`
`6
`be divided up into Virtual LANs (VLANs). The VLAN
`switches 204 are coupled to the Internet 106. In general a
`computing element contains one or two network interfaces
`connected to the VLAN switch. For the sake of simplicity,
`in FIG. 2 all nodes are shown with two network interfaces,
`although some may have less or more network interfaces.
`Many commercial vendors now provide switches supporting
`VLAN functionality. For example, suitable VLAN switches
`are commercially available from Cisco Systems, Inc. and
`Xtreme Networks. Similarly there are a large number of
`commercially available products to construct SANs, includ-
`ing Fibre Channel switches, SCSI-to-Fibre-Channel bridg-
`ing devices, and Network Attached Storage (NAS) devices.
`Control Plane 206 is coupled by a SAN Control path,
`CPU Control path, and VI .AN Control path to SAN switches
`202, CPU5 CPU1, CPU2, .
`.
`. CPUn, and VLAN Switches
`204, respectively.
`Each VSF is composed of a set of VLANs, a set of
`computing elements that are attached to the VLANs, and a
`subset of the storage available on the SAN that is coupled to
`the set of computing elements. The subset of the storage
`available on the SAN is referred to as a SAN Zone and is
`protected by the SAN hardware from access from computing
`elements which are part of other SAN zones. Preferably,
`VLANs that provide non-forgeable port identifiers are used
`to prevent one customer or end user from obtaining access
`to VSF resources of another customer or end user.
`FIG. 3 is a block diagram of an exemplary virtual server
`farm featuring a SAN Zone. Aplurality of Web servers W81,
`W82, etc. are coupled by a first VLAN (VLANl) to a load
`balancer(I.R)/firewall 302. A second VLAN (VI.AN2)
`couples the Internet 106 to the load balancer(LB)/firewall
`302. Each of the Web servers may be selected from among
`CPU1, CPU2, etc., using mechanisms described further
`herein. The Web servers are coupled to a SAN Zone 304,
`which is coupled to one or more storage devices 306a, 3061).
`At any gven point in time, a computing element in the
`computing grid, such as CPUl of FIG. 2, is only connected
`to the set of VLANs and the SAN zone(s) associated with a
`single VSF. A VSF typically is not shared among different
`organizations. The subset of storage on the SAN which
`belongs to a single SAN zone, and the set of VLANs
`associated with it and the computing elements on these
`VLANs define a VSF.
`By controlling the membership of a VLAN and the
`membership of a SAN zone, Control Plane enforces a logical
`partitioning of the computing grid into multiple VSFs.
`Members of one VSF cannot access the computing or
`storage resources of another VSF. Such access restrictions
`are enforced at the hardware level by the VLAN switches,
`and by port—level access control mechanisms (e.g., zoning)
`of SAN hardware such as Fibre Channel switches and edge
`devices such as SCSI to Fibre Channel bridging hardware.
`Computing elements that form part of the computing grid
`are not physically connected to the control ports or inter-
`faces of the VLAN switches and the SAN switches, and
`therefore cannot control the membership of the VLANs or
`SAN zones. Accordingly, the computing elements of the
`computing grid cannot access computing elements not
`located in the VSF in which they are contained.
`Only the computing elements that run the Control Plane
`are physically connected to the control ports or interface of
`the devices in the grid. Devices in the computing grid
`(computers, SAN switches and VLAN switches) can only be
`configured through such control ports or interfaces. This
`provides a simple yet highly secure means of enforcing the
`dynamic partitioning of the computing grid into multiple
`VSFs.
`
`Oracle Exhibit 1006, Page 22
`
`Oracle Exhibit 1006, Page 22
`
`

`

`US 6,779,016 B1
`
`
`
`
`
`7
`Each computing element in a VSF is replaceable by any
`other computing element. The number of computing
`elements, VLANs and SAN zones associated with a given
`VSF may change over time under control of the Control
`Plane.
`In one embodiment, the computing grid includes an Idle
`Pool that comprises large number of computing elements
`that are kept in reserve Computing elements from the Idle
`Pool may be assigned to a particular VSF for reasons such
`as increasing the CPU or memory capacity available to that
`VSF, or to deal with failures of a particu ar computing
`element in a VSF. When the computing elements are con—
`figured as Web servers, the Idle Pool serves as a large “shock
`
`
`absorber” for varying or “bursty” Web tra‘lc loads and
`
`
`related peak processing loads.
`
`The Idle Pool
`is shared between many di erent
`organizations, and therefore it provides economies of scale,
`since no single organization has to pay for the entire cost of
`the Idle Pool. Different organizations can obtain com uting
`elements from the Idle Pool at different times in the cay, as
`needed, thereby enabling each VSF to grow when required
`and shrink when traffic falls down to normal. If many
`different organizations continue to peak at the same time and
`thereby potentially exhaust the capacity of the Idle Pool, the
`Idle Pool can be increased by adding more CPUs and storage
`elements to it (scalability). The capacity of the Idle Pool is
`engineered so as to greatly reduce the probability that, in
`steady state, a particular VSF may not be able to obtain an
`additional computing element from the Idle Pool when it
`needs to.
`FIG. 4A, FIG. 4B, FIG. 4C, and FIG. 4D are block
`diagrams showing successive steps involved in moving a
`computing element in and out of the Idle Pool. Referring
`first to FIG. 4A, assume that the Control Plane has logically
`s)u.
`connected elements of the computing grid into first and .
`second VSFs labeled VSFl, VSFZ. Idle Pool 400 comprises
`a plurality of CPUs 402, one of which is labeled CPUX. In
`FIG. 4B, VSFl has developed a need for an additional
`computing element. Accordingly, the Control Plane moves
`CPUX from Idle Pool 400 to VSFI, as indicated by path
`404.
`In FIG. 4C, VSFl no longer needs CPUX, and therefore
`the Control Plane moves CPUX out of VSFl and back into
`the Idle Pool 400. In FIG. 4D, VSF2 has developed a need
`for an additional computing element. Accordingly, the Con—
`trol Plane moves CPUX from the Idle Pool 400 to VSFZ.
`Thus, over the course of time, as traffic conditions change,
`a single computing element may belong to the Idle Pool
`(FIG. 4A), then be assigned to a particular VSF (FIG. 413),
`then be placed back in the Idle Pool (FIG. 4C), and then
`belong to another VSF (FIG. 4D).
`At each one of these stages, The Control Plane configures
`the LAN switches and SAN switches associated with that
`computing element to be part of the VLANs and SAN zones
`associated with a particular VSF (or the Idle Pool). Accord— .,
`ing to one embodiment,
`in b