`
`
`
`
`
`I IIIII I IIIII IIIIIIII IIIIIIII III 11111 III 11111 11111 JIII811,111)!IIII 11111 JIII811,111)!IIII 11111 11111 111111 111111 1111111111 1111111111
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`US007831739B2
`
`(12) United States Patent
`(12) United States Patent
`(12) United States Patent
`Lambeth et al.
`
`Lambeth et al. Lambeth et al.
`
`(10) Patent No.:
`
`(10) Patent No.: (10) Patent No.:
`(45) Date of Patent:
`
`(45) Date of Patent: (45) Date of Patent:
`
`US 7,831,739 B2
`US 7,831,739 B2
`US 7,831,739 B2
`*Nov. 9, 2010
`
`*Nov. 9, 2010 *Nov. 9, 2010
`
`(54) PARTIAL COPYING OF DATA TO TRANSMIT
`
`(54)
`PARTIAL COPYING OF DATA TO TRANSMIT PARTIAL COPYING OF DATA TO TRANSMIT
`BUFFER FOR VIRTUAL NETWORK DEVICE
`
`BUFFER FOR VIRTUAL NETWORK DEVICE BUFFER FOR VIRTUAL NETWORK DEVICE
`
`(75) Inventors: Walter Andrew Lambeth, San Mateo,
`(75)
`
`
`Inventors: Inventors: Walter Andrew Lambeth, San Mateo, Walter Andrew Lambeth, San Mateo,
`CA (US); Malik Mahalingam,
`
`CA (US); Mallik Mahalingam, CA (US); Mallik Mahalingam,
`Cupertino, CA (US)
`
`Cupertino, CA (US) Cupertino, CA (US)
`
`(73) Assignee: VMware, Inc., Palo Alto, CA (US)
`Assignee: Assignee: VMware, Inc., Palo Alto, CA (US) VMware, Inc., Palo Alto, CA (US)
`
`
`(73)
`(*) Notice:
`Subject to any disclaimer, the term of this
`(* )
`Notice: Notice:
`
`
`Subject to any disclaimer, the term of this Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`
`patent is extended or adjusted under 35 patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`U.S.C. 154(b) by 0 days. U.S.C. 154(b) by 0 days.
`This patent is Subject to a terminal dis
`
`This patent is subject to a terminal dis-This patent is subject to a terminal dis-
`claimer.
`
`claimer. claimer.
`
`(21) Appl. No.: 12/639,590
`(21) Appl. No.: 12/639,590 (21) Appl. No.: 12/639,590
`
`
`(22) Filed:
`(22) Filed:
`(22) Filed:
`
`Dec. 16, 2009
`Dec. 16, 2009
`Dec. 16, 2009
`
`(65)
`(65) (65)
`
`
`Prior Publication Data
`Prior Publication Data Prior Publication Data
`
`US 201O/OO95O45A1
`Apr. 15, 2010
`Apr. 15, 2010 Apr. 15, 2010
`US 2010/0095045 Al US 2010/0095045 Al
`
`
`
`Related U.S. Application Data
`Related U.S. Application Data Related U.S. Application Data
`
`(63) Continuation of application No. 11/565,029, filed on
`
`(63) Continuation of application No. 11/565,029, filed on (63) Continuation of application No. 11/565,029, filed on
`Nov.30, 2006, now Pat. No. 7,657,659.
`
`Nov. 30, 2006, now Pat. No. 7,657,659. Nov. 30, 2006, now Pat. No. 7,657,659.
`
`(51) Int. Cl.
`(51) Int. Cl.
`(51) Int. Cl.
`
`G06F 15/173 GO6F 15/173
`G06F 5/73
`
`(2006.01) (2006.01)
`
`(2006.01)
`
`(52) U.S. Cl. ............................... 709/250; 711/6: 718/1;
`
`
`(52) U.S. Cl. (52) U.S. Cl.
` 709/250; 711/6; 718/1; 709/250; 711/6; 718/1;
`370/470; 370/471; 370/472; 370/473; 370/474
`
`370/470; 370/471; 370/472; 370/473; 370/474 370/470; 370/471; 370/472; 370/473; 370/474
`(58) Field of Classification Search ................. 709/250;
`
`(58) Field of Classification Search (58) Field of Classification Search
`
` 709/250; 709/250;
`711/6: 718/1; 370/470 474
`711/6; 718/1; 370/470-474
`711/6; 718/1; 370/470-474
`See application file for complete search history.
`
`See application file for complete search history. See application file for complete search history.
`References Cited
`References Cited References Cited
`
`
`(56)
`(56) (56)
`
`
`U.S. PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`
`
`6,046,979 A * 6,046,979 A *
`
`4/2000 Bauman 4/2000 Bauman
`370/229 370/229
`6,046,979 A * 4/2000 Bauman ..................... 370,229
`
`7,478,173 B1 * 7,478,173 B1 *
`
`1/2009 Delco 1/2009 Delco
`
`709/250 709/250
`7,478,173 B1* 1/2009 Delco ............
`... TO9,250
`2005/01 14855 A1* 5/2005 Baumberger ................... T18, 1
`
`2005/0114855 Al * 2005/0114855 Al *
`
`5/2005 Baumberger 5/2005 Baumberger
`
`718/1 718/1
`
`2007/0079307 Al * 2007/0079307 Al *
`
`4/2007 Dhawan et al 4/2007 Dhawan et al
`
`718/1 718/1
`
`
`2007/00793.07 A1* 4/2007 Dhawan et al. ................ T18, 1
`* cited by examiner
`
`* cited by examiner * cited by examiner
`Primary Examiner Wing F Chan
`
`Primary Examiner Wing F Chan Primary Examiner Wing F Chan
`Assistant Examiner Hitesh Patel
`
`Assistant Examiner Hitesh Patel Assistant Examiner Hitesh Patel
`(74) Attorney, Agent, or Firm Terry McHugh
`
`(74) Attorney, Agent, or Firm Terry McHugh (74) Attorney, Agent, or Firm Terry McHugh
`
`ABSTRACT
`(57)
`ABSTRACT ABSTRACT
`(57) (57)
`
`
`In a virtualized computer system, a network frame is trans
`In a virtualized computer system, a network frame is trans-
`In a virtualized computer system, a network frame is trans-
`mitted from a virtual machine using a network interface
`
`mitted from a virtual machine using a network interface mitted from a virtual machine using a network interface
`device, possibly through a virtual Switch, by copying only a
`
`device, possibly through a virtual switch, by copying only a device, possibly through a virtual switch, by copying only a
`part of the network frame to the transmit buffers that have
`part of the network frame to the transmit buffers that have
`part of the network frame to the transmit buffers that have
`pre-translated mappings from guest physical addresses to
`pre-translated mappings from guest physical addresses to
`pre-translated mappings from guest physical addresses to
`hypervisor virtual addresses and to machine addresses. The
`
`hypervisor virtual addresses and to machine addresses. The hypervisor virtual addresses and to machine addresses. The
`length of the part of the network frame that is copied to the
`length of the part of the network frame that is copied to the
`length of the part of the network frame that is copied to the
`transmit buffers may be variable.
`transmit buffers may be variable.
`transmit buffers may be variable.
`
`14 Claims, 7 Drawing Sheets
`14 Claims, 7 Drawing Sheets 14 Claims, 7 Drawing Sheets
`
`
`
`
`
`
`
`VM VM
`
`200 200
`
`
`Guest OS Guest OS
`
`220 220
`
`Device Driver
`
`Device Driver Device Driver
`
`272 272
`
`+Partial Copy of Frames
`+Partial Copy of Frames
`
`
`Transmit Buffers Transmit Buffers
`
`402 402
`
`ft++
`t
`ft++
`t
`Partial Copy of Frames
`
`* ♦I I* ♦I I
`
`I Partial Copy of Frames I Partial Copy of Frames
`
`
`VMM VMM
`
`300 300
`
`
`Kernel Kernel
`
`600 600
`
`System Hardware
`
`System Hardware System Hardware
`100
`
`100 100
`
`
`Virtual NIC Virtual NIC
`Wirtual NC
`
`332 332
`
`
`
`• •
`
`
`Virtual Switch Virtual Switch
`Wirtual Switch
`
`602 602
`
`DMA
`DMA
`
`
`NIC NIC
`172-2
`172-2
`
`WIZ, Inc. EXHIBIT - 1080
`WIZ, Inc. v. Orca Security LTD.
`
`WIZ, Inc. EXHIBIT - 1080
`WIZ, Inc. v. Orca Security LTD.
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 9, 2010
`Nov. 9, 2010
`
`Sheet 1 of 7
`Sheet 1 of 7
`
`US 7,831,739 B2
`US 7,831,739 B2
`
`700- 4,
`
`VM
`
`200
`
`260
`
`APPS
`
`t
`GUEST SYSTEM SOFTWARE
`
`-1 GUEST OS
`220
`
`DRVS
`224
`
`VIRTUAL SYSTEM HARDWARE
`
`202
`
`201
`
`VCPUO
`210-0
`
`VCPU1
`210-1
`
`VDEVICE(S)
`270
`
`VMEM
`230
`
`VCPUm
`210-m
`
`VDISK
`240
`
`VM
`200-n
`
`APPS
`
`430
`
`COS
`420
`
`VM M1
`VMM1
`300
`300
`
`DEVICE EMULATORS
`DEVICE EMULATORS
`330
`330
`
`.
`
`.
`
`VMMn
`VMMn
`300-n
`300-n
`
`t
`I
`
`600
`600
`
`SYSTEM HARDWARE
`
`100
`100
`
`
`
`c + SYSTEM HARDWARE
`
`KERNEL
`KERNEL
`
`C C
`CPU(S) MEM Disk1 DEyCES
`DEVICES
`MEM
`CPU(S)
`[DISK
`170
`140
`170
`130
`110
`130
`110
`140
`
`FIG. 1
`FIG. 1
`(PRIOR ART)
`(PRIOR ART)
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 9, 2010
`Nov. 9, 2010
`
`Sheet 2 of 7
`Sheet 2 of 7
`
`US 7,831,739 B2
`US 7,831,739 B2
`
`
`
`VM
`
`260
`
`APPS
`
`200
`
`--- 220
`
`GUEST
`OS
`
`‘ 292
`
`GUEST OS
`GUEST OS
`PAGE
`PAGE
`TABLE
`TABLE
`
`GVPN-►
`GPPN
`GVPND GPPN
`
`201
`VIRTUAL
`VIRTUAL
`SYSTEM
`SYSTEM
`HW
`HW
`
`A
`
`y
`
`1_
`
`VMM
`
`300
`
`350
`
`MEMORY
`MEMORY
`MANAGEMENT GPPN* PPN
`MANAGEMENT GPPN D PPN
`MODULE
`MODULE
`
`392
`
`C
`SHADOW
`SHADOW
`PAGE GVPN-> PPN
`PAGE GVPN- o- PPN
`TABLE
`TABLE
`
`1
`i
`
`7 -100
`
`194
`194
`
`SYSTEM
`SYSTEM
`HARDWARE
`HARDWARE
`
`TLB
`
`GVPN->PPN
`GVPN-DPPN
`
`FIG. 2
`FIG. 2
`(PRIOR ART)
`(PRIOR ART)
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 9, 2010
`Nov. 9, 2010
`
`Sheet 3 of 7
`Sheet 3 of 7
`
`US 7,831,739 B2
`US 7,831,739 B2
`
`VM
`VM
`200-2
`200-2
`
`VM
`VM
`200-3
`200-3
`
`VM
`VM
`200-1
`200-1
`
`VMM
`VMM
`300-1
`300-1
`
`Kernel
`600
`600
`
`VMM
`VMM
`300-2
`300-2
`
`VMM
`VMM
`300-3
`
`300-3 /Virtual Switch
`
`Virtual SWitch
`602
`
`
`
`System Hardware
`System Hardware
`100
`100
`
`NIC
`172-1
`
`NIC
`172-2
`
`NIC
`172-3
`
`FIG. 3
`FIG. 3
`(PRIOR ART)
`(PRIOR ART)
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 9, 2010
`Nov. 9, 2010
`
`Sheet 4 of 7
`Sheet 4 of 7
`
`US 7,831,739 B2
`US 7,831,739 B2
`
`VM
`200
`
`Guest OS
`220
`
`Device Driver
`272
`
`
`
`I
`
`Partial Copy of Frames
`Partial Copy of Frames
`
`Transmit Buffers
`Transmit Buffers
`402
`402
`
`t
`7
`
`++++
`Partial Copy of Frames
`Partial Copy of Frames
`f I
`
`VMM
`300
`
`Virtual NIC
`Virtual NC
`332
`332
`
`Kernel
`600
`
`
`
`System Hardware
`System Hardware
`100
`100
`
`+
`
`V
`
`Virtual Switch
`Virtual SWitch
`602
`602
`
`DMA
`DMA
`
`NIC
`172-2
`
`FIG. 4
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 9, 2010
`Nov. 9, 2010
`
`Sheet 5 Of 7
`Sheet 5 of 7
`
`US 7,831,739 B2
`US 7,831,739 B2
`
`
`
`DEVICE DRIVER INITIALIZES
`DEVICE DRIVER INITIALIZES
`TRANSMIT BUFFERS
`TRANSMIT BUFFERS
`502
`502
`
`CALL INTO VIRTUALIZATION SOFTWARE
`CALL INTO WIRTUALIZATION SOFTWARE
`WITH ARRAY OF PA FOR PRE-
`WITH ARRAY OF PA FOR PRE
`TRANSLATION IN TRANSMIT BUFFERS
`TRANSLATION IN TRANSMIT BUFFERS
`504
`504
`
`VIRTUALIZATION SOFTWARE MAPS PA TO
`VIRTUALIZATION SOFTWARE MAPS PATO
`VA AND PA TO MA FOR
`VA AND PATO MA FOR
`EACH PA ENTRY OF ARRAY
`EACH PA ENTRY OF ARRAY
`506
`506
`
`DEVICE DRIVER CALLS VIRTUALIZATION
`DEVICE DRIVER CALLSVIRTUALIZATION
`SOFTWARE TO RELEASE PA ARRAY
`SOFTWARE TO RELEASE PA ARRAY
`508
`508
`
`TRANSMIT BUFFERS ARE
`TRANSMIT BUFFERS ARE
`PRE-TRANSLATED WITH
`PRE-TRANSLATED WITH
`PA TO VA AND PA TO MA MAPPINGS
`PATO VA AND PATO MA MAPPINGS
`510
`510
`
`FIG. 5
`FIG. 5
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 9, 2010
`Nov. 9, 2010
`
`Sheet 6 of 7
`Sheet 6 of 7
`
`US 7,831,739 B2
`US 7,831,739 B2
`
`
`
`GUEST OS REOUESTS
`GUEST OS REQUESTS
`TRANSMISSION OF FRAME
`TRANSMISSION OF FRAME
`TO DEVICE DRIVER
`TO DEVICE DRIVER
`602
`602
`
`DEVICE DRIVER EXAMINES
`DEVICE DRIVEREXAMINES
`FRAME HEADERS AND COMPUTES
`FRAME HEADERS AND COMPUTES
`LENGTH (NBYTES) OF PART OF FRAME
`LENGTH (N BYTES) OF PART OF FRAME
`TO BE COPIED
`TO BE COPED
`604
`604
`
`DEVICE DRIVER COPIES N BYTES OF
`DEVICE DRIVER COPES N BYTES OF
`FRAME TO TRANSMIT BUFFER
`FRAME TO TRANSMIT BUFFER
`606
`606
`
`VIRTUALIZATION SOFTWARE OBTAINS
`VIRTUALIZATION SOFTWARE OBTAINS
`PRE-TRANSLATED VA AND MA
`PRE-TRANSLATED WA AND MA
`CORRESPONDING TO PA ASSOCATED
`CORRESPONDING TO PA ASSOCIATED
`WITH COPIED PART OF FRAME
`WITH COPIED PART OF FRAME
`608
`608
`
`SECOND COPY OF FRAME IS MADE TO
`SECOND COPY OF FRAME IS MADE TO
`ANOTHER BUFFER NOT ACCESSIBLE TO
`ANOTHER BUFFER NOT ACCESSIBLE TO
`THE GUEST OS
`THE GUEST OS
`609
`609
`
`PA ASSOCIATED WITH
`PA ASSOCATED WITH
`REMAINING PART OF FRAME
`REMAINING PART OF FRAME
`TRANSLATED AND MAPPED TO MA
`TRANSLATED AND MAPPED TO MA
`610
`610
`
`NIC DRIVER USES PRE-TRANSLATED MA
`NC DRIVER USES PRE-TRANSLATED MA
`OF COPIED PART OF FRAME
`OF COPIED PART OF FRAME
`AND TRANSLATED MA OF REMAINING
`AND TRANSLATED MA OF REMAINING
`PART OF FRAME TO DMA FRAME
`PART OF FRAME TO DMA FRAME
`612
`612
`
`FIG. 6
`FIG. 6
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 9, 2010
`Nov. 9, 2010
`
`Sheet 7 Of 7
`Sheet 7 of 7
`
`US 7,831,739 B2
`US 7,831,739 B2
`
`700
`700
`
`vc)
`
`
`
`Ethernet Header
`Ethernet Header
`Section
`Section
`702
`702
`
`TCP/UDP Section
`P Header Section TCP/UDP Section
`IP Header Section
`704
`706
`
`Remaining Section
`Remaining Section
`708
`708
`
`FIG. 7
`FIG. 7
`
`

`

`CROSS REFERENCE TO RELATED
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`APPLICATIONS
`
`BACKGROUND OF THE INVENTION
`BACKGROUND OF THE INVENTION
`
`1
`1.
`PARTIAL COPYING OF DATA TO TRANSMIT
`PARTIAL COPYING OF DATA TO TRANSMIT
`BUFFER FOR VIRTUAL NETWORK DEVICE
`BUFFER FOR VIRTUAL NETWORK DEVICE
`
`US 7,831,739 B2
`US 7,831,739 B2
`
`This application is a continuation and claims benefit of
`This application is a continuation and claims benefit of
`U.S. patent application Ser. No. 11/565,029, now U.S. Pat.
`U.S. patent application Ser. No. 11/565,029, now U.S. Pat.
`No. 7,657,659, which was filed Nov.30, 2006.
`No. 7,657,659, which was filed Nov. 30, 2006.
`
`2
`2
`example, illustrates multiple virtual processors 210-0,
`example, illustrates multiple virtual processors 210-0,
`210-1,..., 210-m (VCPU0, VCPU1,..., VCPUm) within the
`210-1,
`, 210-m (VCPU0, VCPUL
`, VCPUm) within the
`VM2OO.
`VM 200.
`Yet another configuration is found in a so-called “multi
`Yet another configuration is found in a so-called "multi-
`core' architecture, in which more than one physical CPU is
`5 core" architecture, in which more than one physical CPU is
`fabricated on a single chip, with its own set of functional units
`fabricated on a single chip, with its own set of functional units
`(such as a floating-point unit and an arithmetic/logic unit
`(such as a floating-point unit and an arithmetic/logic unit
`ALU), and can execute threads independently; multi-core
`ALU), and can execute threads independently; multi-core
`processors typically share only very limited resources. Such
`processors typically share only very limited resources, such
`as some cache. Still another technique that provides for
`10 as some cache. Still another technique that provides for
`10
`simultaneous execution of multiple threads is referred to as
`simultaneous execution of multiple threads is referred to as
`'simultaneous multi-threading.” in which more than one logi
`"simultaneous multi-threading," in which more than one logi-
`cal CPU (hardware thread) operates simultaneously on a
`cal CPU (hardware thread) operates simultaneously on a
`1. Field of the Invention
`1. Field of the Invention
`single chip, but in which the logical CPUs flexibly share some
`single chip, but in which the logical CPUs flexibly share some
`This invention relates generally to a virtualized computer
`resource Such as caches, buffers, functional units, etc. This
`15 resource such as caches, buffers, functional units, etc. This
`This invention relates generally to a virtualized computer
`15
`invention may be used regardless of the type physical and/
`system and, in particular, to a method and system for trans
`invention may be used regardless of the type physical and/
`system and, in particular, to a method and system for trans-
`or logical—or number of processors included in a VM.
`mitting data from virtual machines to a network interface
`or logical or number of processors included in a VM.
`mitting data from virtual machines to a network interface
`If the VM 200 is properly designed, applications 260 run
`device via a virtual Switch by partial copying of the data to
`If the VM 200 is properly designed, applications 260 run-
`device via a virtual switch by partial copying of the data to
`transmit buffers with pre-mapped address translations.
`ning on the VM will function as they would if run on a “real
`ning on the VM will function as they would if run on a "real"
`transmit buffers with pre-mapped address translations.
`computer, even though the applications are running at least
`20 computer, even though the applications are running at least
`2. Description of the Related Art
`2. Description of the Related Art
`partially indirectly, that is via the guest OS 220 and virtual
`partially indirectly, that is via the guest OS 220 and virtual
`The advantages of virtual machine technology have
`The advantages of virtual machine technology have
`processor(s). Executable files will be accessed by the guest
`processor(s). Executable files will be accessed by the guest
`become widely recognized. Among these advantages is the
`become widely recognized. Among these advantages is the
`OS from the virtual disk 240 or virtual memory 230, which
`OS from the virtual disk 240 or virtual memory 230, which
`ability to run multiple virtual machines on a single host plat
`ability to run multiple virtual machines on a single host plat-
`will be portions of the actual physical disk 140 or memory
`will be portions of the actual physical disk 140 or memory
`form. This makes better use of the capacity of the hardware,
`form. This makes better use of the capacity of the hardware,
`130 allocated to that VM. Once an application is installed
`25 130 allocated to that VM. Once an application is installed
`25
`while still ensuring that each user enjoys the features of a
`while still ensuring that each user enjoys the features of a
`within the VM, the guest OS retrieves files from the virtual
`within the VM, the guest OS retrieves files from the virtual
`“complete' computer. Depending on how it is implemented,
`"complete" computer. Depending on how it is implemented,
`disk just as if the files had been pre-stored as the result of a
`disk just as if the files had been pre-stored as the result of a
`virtualization can also provide greater security, since the Vir
`virtualization can also provide greater security, since the vir-
`conventional installation of the application. The design and
`conventional installation of the application. The design and
`tualization can isolate potentially unstable or unsafe Software
`tualization can isolate potentially unstable or unsafe software
`operation of virtual machines are well known in the field of
`operation of virtual machines are well known in the field of
`so that it cannot adversely affect the hardware state or system
`so that it cannot adversely affect the hardware state or system
`computer Science.
`30 computer science.
`30
`files required for running the physical (as opposed to virtual)
`files required for running the physical (as opposed to virtual)
`Some interface is generally required between the guest
`Some interface is generally required between the guest
`hardware.
`hardware.
`software within a VM and the various hardware components
`software within a VM and the various hardware components
`As is well known in the field of computer science, a virtual
`As is well known in the field of computer science, a virtual
`and devices in the underlying hardware platform. This inter
`and devices in the underlying hardware platform. This inter-
`machine (VM) is an abstraction—a “virtualization' of an
`machine (VM) is an abstraction a "virtualization"—of an
`face which may be referred to generally as “virtualization
`face which may be referred to generally as "virtualization
`actual physical computer system. FIG. 1 shows one possible
`actual physical computer system. FIG. 1 shows one possible
`Software' may include one or more software components
`35 software" may include one or more software components
`35
`arrangement of a computer system 700 that implements vir
`arrangement of a computer system 700 that implements vir-
`and/or layers, possibly including one or more of the Software
`and/or layers, possibly including one or more of the software
`tualization. A virtual machine (VM) or "guest’ 200 is
`tualization. A virtual machine (VM) or "guest" 200 is
`components known in the field of virtual machine technology
`components known in the field of virtual machine technology
`installed on a “host platform, or simply “host,” which will
`installed on a "host platform," or simply "host," which will
`as “virtual machine monitors” (VMMs), “hypervisors.” or
`as "virtual machine monitors" (VMMs), "hypervisors," or
`include system hardware, that is, a hardware platform 100,
`include system hardware, that is, a hardware platform 100,
`virtualization “kernels.” Because virtualization terminology
`virtualization "kernels." Because virtualization terminology
`and one or more layers or co-resident components comprising
`and one or more layers or co-resident components comprising
`has evolved over time and has not yet become fully standard
`40 has evolved over time and has not yet become fully standard-
`40
`system-level Software. Such as an operating system or similar
`system-level software, such as an operating system or similar
`ized, these terms do not always provide clear distinctions
`ized, these terms do not always provide clear distinctions
`kernel, or a virtual machine monitor or hypervisor (see
`kernel, or a virtual machine monitor or hypervisor (see
`between the software layers and components to which they
`between the software layers and components to which they
`below), or some combination of these. The system hardware
`below), or some combination of these. The system hardware
`refer. For example, “hypervisor” is often used to describe
`refer. For example, "hypervisor" is often used to describe
`typically includes one or more processors 110, memory 130,
`typically includes one or more processors 110, memory 130,
`both a VMM and a kernel together, either as separate but
`both a VMM and a kernel together, either as separate but
`Some form of mass storage 140, and various other devices
`some form of mass storage 140, and various other devices
`cooperating components or with one or more VMMs incor
`45 cooperating components or with one or more VMMs incor-
`45
`170.
`170.
`porated wholly or partially into the kernel itself, which will be
`porated wholly or partially into the kernel itself, which will be
`Each VM 200 will typically have both virtual system hard
`Each VM 200 will typically have both virtual system hard-
`the meaning of hypervisor as used herein; however, "hyper
`the meaning of hypervisor as used herein; however, "hyper-
`ware 201 and guest system software 202. The virtual system
`ware 201 and guest system software 202. The virtual system
`visor” is sometimes used instead to mean some variant of a
`visor" is sometimes used instead to mean some variant of a
`hardware typically includes at least one virtual CPU, virtual
`hardware typically includes at least one virtual CPU, virtual
`VMM alone, which interfaces with some other software
`VMM alone, which interfaces with some other software
`memory 230, at least one virtual disk 240, and one or more
`memory 230, at least one virtual disk 240, and one or more
`layer(s) or component(s) to support the virtualization. More
`50 layer(s) or component(s) to support the virtualization. More-
`50
`virtual devices 270. Note that a disk virtual or physical is
`virtual devices 270. Note that a disk virtual or physical
`is
`over, in Some systems, some virtualization code is included in
`over, in some systems, some virtualization code is included in
`also a “device.” but is usually considered separately because
`also a "device," but is usually considered separately because
`at least one “superior VM to facilitate the operations of other
`at least one "superior" VM to facilitate the operations of other
`of the important role of the disk. All of the virtual hardware
`of the important role of the disk. All of the virtual hardware
`VMs. Furthermore, specific software support forVMs may be
`VMs. Furthermore, specific software support forVMs may be
`components of the VM may be implemented in software
`components of the VM may be implemented in software
`included in the host OS itself. Unless otherwise indicated, the
`included in the host OS itself. Unless otherwise indicated, the
`using known techniques to emulate the corresponding physi
`using known techniques to emulate the corresponding physi-
`invention described below may be used in virtualized com
`55 invention described below may be used in virtualized com-
`55
`cal components. The guest system software includes a guest
`puter systems having any type or configuration of virtualiza
`cal components. The guest system software includes a guest
`puter systems having any type or configuration of virtualiza-
`operating system (OS) 220 and drivers 224 as needed for the
`operating system (OS) 220 and drivers 224 as needed for the
`tion software.
`tion software.
`various virtual devices 270.
`various virtual devices 270.
`Moreover, FIG. 1 shows virtual machine monitors that
`Moreover, FIG. 1 shows virtual machine monitors that
`appear as separate entities from other components of the
`Note that a single VM may be configured with more than
`appear as separate entities from other components of the
`Note that a single VM may be configured with more than
`one virtualized processor. To permit computer systems to
`virtualization software. Furthermore, some software compo
`60 virtualization software. Furthermore, some software compo-
`one virtualized processor. To permit computer systems to
`60
`scale to larger numbers of concurrent threads, systems with
`nents used to implement one illustrated embodiment of the
`nents used to implement one illustrated embodiment of the
`scale to larger numbers of concurrent threads, systems with
`multiple CPUs have been developed. These symmetric multi
`invention are shown and described as being within a “virtu
`invention are shown and described as being within a "virtu-
`multiple CPUs have been developed. These symmetric multi-
`processor (SMP) systems are available as extensions of the
`alization layer
`located logically between all virtual
`alization layer" located logically between all virtual
`processor (SMP) systems are available as extensions of the
`machines and the underlying hardware platform and/or sys
`PC platform and from other vendors. Essentially, an SMP
`machines and the underlying hardware platform and/or sys-
`PC platform and from other vendors. Essentially, an SMP
`system is a hardware platform that connects multiple proces
`tem-level host software. This virtualization layer can be con
`65 tem-level host software. This virtualization layer can be con-
`system is a hardware platform that connects multiple proces-
`65
`sidered part of the overall virtualization software, although it
`sors to a shared main memory and shared I/O devices. Virtual
`sors to a shared main memory and shared I/O devices. Virtual
`sidered part of the overall virtualization software, although it
`would be possible to implement at least part of this layer in
`machines may also be configured as SMPVMs. FIG. 1, for
`machines may also be configured as SMP VMs. FIG. 1, for
`would be possible to implement at least part of this layer in
`
`

`

`US 7,831,739 B2
`US 7,831,739 B2
`
`5
`
`3
`4
`4
`3
`operations, alongside and sometimes at the request of the
`specialized hardware. The illustrated embodiments are given
`operations, alongside and sometimes at the request of the
`specialized hardware. The illustrated embodiments are given
`only for the sake of simplicity and clarity and by way of
`VMM. The Workstation product of VMware, Inc., of Palo
`VMM. The Workstation product of VMware, Inc., of Palo
`only for the sake of simplicity and clarity and by way of
`Alto, Calif., is an example of a hosted, virtualized computer
`illustration—as mentioned above, the distinctions are not
`Alto, Calif., is an example of a hosted, virtualized computer
`illustration as mentioned above, the distinctions are not
`system, which is also explained in U.S. Pat. No. 6,496,847
`always so clear-cut. Again, unless otherwise indicated or
`system, which is also explained in U.S. Pat. No. 6,496,847
`always so clear-cut. Again, unless otherwise indicated or
`(Bugnion, et al., “System and Method for Virtualizing Com
`apparent from the description, it is to be assumed that the
`(Bugnion, et al., "System and Method for Virtualizing Com-
`apparent from the description, it is to be assumed that the
`puter Systems.” 17 Dec. 2002).
`invention can be implemented anywhere within the overall
`puter Systems," 17 Dec. 2002).
`invention can be implemented anywhere within the overall
`As illustrated in FIG. 1, in many cases, it may be beneficial
`structure of the virtualization Software, and even in Systems
`As illustrated in FIG. 1, in many cases, it may be beneficial
`structure of the virtualization software, and even in systems
`that provide specific hardware support for virtualization.
`to deploy VMMs on top of a software layer—a kernel 600–
`to deploy VMMs on top of a software layer a kernel 600
`that provide specific hardware support for virtualization.
`constructed specifically to provide efficient support for the
`The various virtualized hardware components in the VM,
`constructed specifically to provide efficient support for the
`The various virtualized hardware components in the VM,
`VMs. This configuration is frequently referred to as being
`such as the virtual CPU(s) 210-0, 210-1, . . . , 210-m, the
`10 VMs. This configuration is frequently referred to as being
`such as the virtual CPU(s) 210-0, 210-1, . . . , 210-m, the
`10
`“non-hosted. Compared with a system in which VMMs run
`virtual memory 230, the virtual disk 240, and the virtual
`"non-hosted." Compared with a system in which VMMs run
`virtual memory 230, the virtual disk 240, and the virtual
`directly on the hardware platform, use of a kernel offers
`device(s) 270, are shown as being part of the VM 200 for the
`directly on the hardware platform, use of a kernel offers
`device(s) 270, are shown as being part of the VM 200 for the
`sake of conceptual simplicity. In actuality, these “compo
`greater modularity and facilitates provision of services (for
`greater modularity and facilitates provision of services (for
`sake of conceptual simplicity. In actuality, these "compo-
`example, resource management) that extend across multiple
`nents' are usually implemented as software emulations 330
`example, resource management) that extend across multiple
`nents" are usually implemented as software emulations 330
`virtual machines. Compared with a hosted deployment, a
`included in a VMM300. One advantage of such an arrange
`15 virtual machines. Compared with a hosted deployment, a
`included in a VMM 300. One advantage of such an arrange-
`15
`kernel may offer greater performance because it can be co
`ment is that the VMM may (but need not) be set up to expose
`kernel may offer greater performance because it can be co-
`ment is that the VMM may (but need not) be set up to expose
`developed with the VMM and be optimized for the charac
`“generic' devices, which facilitate VM migration and hard
`developed with the VMM and be optimized for the charac-
`"generic" devices, which facilitate VM migration and hard-
`ware platform-independence.
`teristics of a workload consisting primarily of VMS/VMMs.
`teristics of a workload consisting primarily of VMs/VMMs.
`ware platform-independence.
`Different systems may implement virtualization to differ
`The kernel 600 also handles any other applications running on
`The kernel 600 also handles any other applications running on
`Different systems may implement virtualization to differ-
`ent degrees—"virtualization' generally relates to a spectrum
`it that can be separately scheduled, as well as a console
`20 it that can be separately scheduled, as well as a console
`ent degrees "virtualization" generally relates to a spectrum
`operating system that, in Some architectures, is used to boot
`of definitions rather than to a bright line, and often reflects a
`operating system that, in some architectures, is used to boot
`of definitions rather than to a bright line, and often reflects a
`design choice with respect to a trade-off between speed and
`the system and facilitate certain user interactions with the
`the system and facilitate certain user interactions with the
`design choice with respect to a trade-off between speed and
`efficiency on the one hand and isolation and universality on
`virtualization software.
`virtualization software.
`efficiency on the one hand and isolation and universality on
`the other hand. For example, “full virtualization' is some
`Note that the kernel 600 is not the same as the kernel that
`Note that the kernel 600 is not the same as the kernel that
`the other hand. For example, "full virtualization" is some-
`times used to denote a system in which no software compo
`will be within the guest OS 220 as is well known, every
`25 will be within the guest OS 220 as is well known, every
`times used to denote a system in which no software compo-
`25
`operating system has its own kernel. Note also that the kernel
`nents of any form are included in the guest other than those
`operating system has its own kernel. Note also that the kernel
`nents of any form are included in the guest other than those
`600 is part of the “host” platform of the VM/VMMas defined
`that would be found in a non-virtualized computer; thus, the
`600 is part of the "host" platform of the VM/VMM as defined
`that would be found in a non-virtualized computer; thus, the
`above even though the configuration shown in FIG. 1 is com
`guest OS could be an off-the-shelf, commercially available
`above even though the configuration shown in FIG. 1 is com-
`guest OS could be an off-the-