`Virtualization Technology
`
`Jack Lo
`Sr. Director, R&D
`
`Columbia Ex 2043-1
`Symantec v Columbia
`IPR2015-00375
`
`
`
`This presentation may contain VMware
`confidential information.
`
`Copyright © 2005 VMware, Inc. All rights reserved. All other
`marks and names mentioned herein may be trademarks of their respective
`companies.
`
`Columbia Ex 2043-2
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Overview
`
`(cid:131) Emerging technologies that impact CPU
`virtualization
`(cid:131) Hardware assist (VT-x/Pacifica)
`(cid:131) 64-bit computing
`(cid:131) OS assist (paravirtualization)
`
`(cid:131) Today’s talk:
`(cid:131) Share our perspective on emerging
`technologies
`
`Columbia Ex 2043-3
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Agenda
`(cid:131) CPU virtualization technology overview
`(cid:131) Virtualizing the x86 architecture
`(cid:131) Trend No. 1: Hardware assist
`(cid:131) VT-x and Pacifica
`(cid:131) Trend No. 2: 64-bit computing
`(cid:131) Benefits of 64-bit architecture
`(cid:131) 64-bit guest support
`(cid:131) Trend No. 3: OS assist
`(cid:131) VMware and paravirtualization
`
`Columbia Ex 2043-4
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Full Virtualization Software Stack
`Third-party Solutions
`
`Management
`and
`Distributed
`Virtualization
`Services
`
`VMotion
`Provisioning
`Backup
`DRS
`DAS
`Distributed
`Services
`
`VirtualCenter
`
`SDK / VirtualCenter Agent
`
`VMX
`
`VMX VMX
`
`VMX
`
`VM
`
`VM VM VM
`
`Third-
`Party
`Agents
`
`Service
`Console
`
`I/O Stack
`Device Drivers
`
`Resource
`Management
`CPU Scheduling
`Memory Scheduling
`Storage Bandwidth
`Network Bandwidth
`ESX Server
`
`Hardware
`
`Virtual
`Machine
`Monitor
`
`Enterprise
`Class
`Virtualization
`Functionality
`
`VMM
`
`VMM VMM
`
`VMM
`
`Distributed
`Virtual Machine
`File System
`
`Virtual NIC
`and
`Switch
`
`Storage Stack Network Stack
`
`Device Drivers
`
`VMkernel Hardware Interface
`
`Columbia Ex 2043-5
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Today’s Focus
`Third-party Solutions
`
`Management
`and
`Distributed
`Virtualization
`Services
`
`VMotion
`Provisioning
`Backup
`DRS
`DAS
`Distributed
`Services
`
`VirtualCenter
`
`SDK / VirtualCenter Agent
`
`VMX
`
`VMX VMX
`
`VMX
`
`VM
`
`VM VM
`
`VM
`
`Third-
`Party
`Agents
`
`Service
`Console
`
`I/O Stack
`Device Drivers
`
`Resource
`Management
`CPU Scheduling
`Memory Scheduling
`Storage Bandwidth
`Network Bandwidth
`ESX Server
`
`Hardware
`
`Virtual
`Machine
`Monitor
`
`Enterprise
`Class
`Virtualization
`Functionality
`
`VMM
`
`VMM VMM
`
`VMM
`
`Distributed
`VM File System
`
`Virtual NIC &
`Switch
`
`Storage Stack Network Stack
`
`Device Drivers
`
`VMkernel Hardware Interface
`
`Columbia Ex 2043-6
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Virtualization SW Terminology
`
`VMM
`
`VMM
`
`VMM
`
`Enhanced
`Functionality
`
`Base Functionality (e.g. scheduling)
`Hypervisor
`(cid:131) Virtual Machine Monitor (VMM)
`(cid:131) SW component that implements virtual machine hardware abstraction
`(cid:131) Responsible for running the guest OS
`(cid:131) Hypervisor
`(cid:131) Software responsible for hosting and managing virtual machines
`(cid:131) Run directly on the hardware
`(cid:131) Functionality varies greatly with architecture and implementation
`
`Columbia Ex 2043-7
`Symantec v Columbia
`IPR2015-00375
`
`
`
`CPU Virtualization
`(cid:131) Three components to classical virtualization techniques
`(cid:131) Many virtualization technologies focus on handling privileged
`instructions
`
`Privileged instruction
`virtualization
`
`Memory virtualization
`
`De-privileging or ring
`compression to handle privileged
`instructions
`Memory partitioning and
`allocation of physical memory
`
`Device and I/O
`virtualization
`
`Routing I/O requests between
`virtual devices and physical
`hardware
`
`Columbia Ex 2043-8
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Apps
`
`Ring 3
`
`Guest OS
`
`Ring 0
`
`Handling Privileged Instructions
`(cid:131) In traditional systems
`(cid:131)OS runs in privileged mode
`(cid:131)OS “owns” the hardware
`(cid:131)Application code has less privilege
`(cid:131) VMM needs highest privilege level
`for isolation and performance
`(cid:131) Traditional VMM relies on “ring
`compression” or “de-privileging”
`(cid:131)Run privileged guest OS code at user-level
`(cid:131)Privileged instructions trap, and emulated by VMM
`
`Guest OS
`
`Apps
`
`Ring 3
`
`VMM
`
`Ring 0
`
`Columbia Ex 2043-9
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Virtualizing x86 Architecture
`(cid:131)De-privileging not possible with x86!
`(cid:131)Some privileged instructions have different
`semantics at user-level: “non-virtualizable
`instructions”
`(cid:131)VMware uses direct execution and
`binary translation (BT)
`(cid:131)BT for handling privileged code
`(cid:131)Direct execution of user-level code for
`performance
`(cid:131)Any unmodified x86 OS can run in virtual
`machine
`
`Columbia Ex 2043-10
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Protecting the VMM
`(cid:131) Need to protect VMM and ensure isolation
`(cid:131) Protect virtual machines from each other
`(cid:131) Protect VMM from virtual machines
`(cid:131) VMware relies on segmentation hardware
`to protect the VMM
`(cid:131) VMM lives at top of guest address space
`(cid:131) Segment limit checks catch writes to VMM area
`VMM
`
`0
`
`4GB
`
`Columbia Ex 2043-11
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Agenda
`(cid:131) CPU virtualization technology overview
`(cid:131) Virtualizing the x86 architecture
`(cid:131) Trend No. 1: Hardware assist
`(cid:131) Trend No. 2: 64-bit computing
`(cid:131) Trend No. 3: OS assist
`
`Columbia Ex 2043-12
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Trend No. 1: Hardware Assist
`(cid:131) CPU vendors are embracing virtualization
`(cid:131)Intel Virtualization Technology (VT-x)
`(cid:131)AMD Pacifica
`(cid:131) These CPU technologies are a series of
`enhancements to aid virtualization SW
`(cid:131)Initially focused on handling non-virtualizable
`instructions
`(cid:131)Use a trap-and-emulate model
`(cid:131)Alternative to using binary translation
`(cid:131) But hardware assist does not eliminate
`need for VMware technology
`
`Columbia Ex 2043-13
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Non-root mode
`
`Root mode
`
`Apps
`
`Apps
`
`Ring 3
`
`Guest OS
`
`Guest OS
`
`Ring 0
`
`VM
`exit
`
`VM
`enter
`
`Virtual Machine Monitor (VMM)
`
`VT-x/Pacifica Overview
`(cid:131) Key feature is new CPU
`execution mode (root mode)
`(cid:131) VMM executes in
`root mode
`(cid:131) Allows x86 virtualization
`without binary translation
`or paravirtualization
`(cid:131) Guest state stored in
`Virtual Machine Control
`Structures (VT-x) or Virtual
`Machine Control Block
`(Pacifica)
`
`Columbia Ex 2043-14
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Limitations of Hardware Assist
`(cid:131) Initial VT-x/Pacifica hardware does not include all
`components of CPU virtualization solution
`(cid:131) VT-x requires small emulator for real mode code
`(cid:131) Memory virtualization support lacking
`(cid:131) Not in VT-x; implementation-dependent for Pacifica
`(cid:131) Memory virtualization is key to performance!
`(cid:131) No device virtualization support
`Hardware
`Assist
`Yes
`No
`No
`
`Privileged instructions
`Memory virtualization
`Device and I/O virtualization
`
`Columbia Ex 2043-15
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Future of Hardware Assist
`(cid:131) CPU vendors will add more hardware capabilities
`in future
`(cid:131) Memory virtualization (Nested paging, EPT)
`(cid:131) VMware software will evolve to incorporate
`support for these new technologies
`(cid:131) Adopt technologies as they enable new capabilities
`Hardware Solution
`VT-x, Pacifica
`
`Extended Page
`Tables/Nested Paging
`Intelligent Devices
`
`Privileged
`instructions
`Memory virtualization
`
`Devices and I/O
`
`Columbia Ex 2043-16
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Trend No. 2: 64-bit Computing
`(cid:131) Progression of the x86 architecture
`(cid:131) 16-bit: 8086/8088 (1978)
`(cid:131) 32-bit: 80386 (1985)
`(cid:131) 64-bit: x86-64 (2003): a.k.a. AMD64, x64
`(cid:131) x86-64 architecture brings 64-bit
`computing to industry-standard
`systems
`(cid:131) Provides compatibility mode to run 32-bit
`x86 applications
`(cid:131) Extensions to x86 architecture
`
`Columbia Ex 2043-17
`Symantec v Columbia
`IPR2015-00375
`
`
`
`64-bit Transition Has Already Begun
`(cid:131) Apps exhausting limits of 32-bit address space
`(cid:131) Consuming 1 bit of address space / year
`(cid:131) Databases, Java app servers, other threaded
`applications
`(cid:131) Most new CPUs are 64-bit enabled
`(cid:131) AMD64, EM64T
`(cid:131) Major OSes have been ported
`(cid:131) Windows, Linux, Solaris 10, etc.
`(cid:131) Applications are being ported
`(cid:131) Databases, app servers, development tools,
`games, etc.
`
`Columbia Ex 2043-18
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Virtualization And x86-64
`(cid:131) Potential questions about 64-bit transition
`(cid:131) Do my apps run in 64-bit OS?
`(cid:131) Have drivers been ported?
`(cid:131) Are the 64-bit OSes robust?
`(cid:131) The solution: virtualization!
`(cid:131) Great aid for 64-bit transition
`(cid:131) Easy way to evaluate new
`OSes
`(cid:131) Can run 64-bit guest OSes
`on 32-bit host OS on 64-bit hardware)!
`
`32-bit or 64-bit
`Host OS
`
`64-bit Hardware
`
`Applications
`
`32-
`bit
`VM
`
`64-
`bit
`VM
`
`VMM
`
`VMM64
`
`Columbia Ex 2043-19
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Challenges of Virtualizing x86-64
`(cid:131) Initial AMD64 architecture did not include
`segmentation in 64-bit mode
`(cid:131) Segmentation also missing from EM64T
`How do we protect the VMM?
`(cid:131) 64-bit guest support requires additional
`hardware assistance
`(cid:131) Segment limit checks available in 64-bit mode
`on newer AMD processors
`(cid:131) VT-x can be used to protect the VMM on EM64T
`(cid:131) Requires trap-and-emulate approach instead of BT
`
`Columbia Ex 2043-20
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Flexible VMM Architecture
`(cid:131) Flexible “multi-mode” VMM architecture
`(cid:131) Separate VMM per virtual
`machine
`(cid:131) 32-bit: BT VMM
`(cid:131) 64-bit: BT or VT/Pacifica
`VMM depending on
`hardware
`(cid:131) Select mode that
`achieves best workload-
`specific performance
`(cid:131) Same VMM architecture for ESX Server,
`GSX Server, Workstation and ACE
`
`VM
`
`VM
`
`VM
`
`BT/VT
`VMM64
`
`BT
`VMM32
`
`BT/VT
`VMM64
`
`. . .
`
`. . .
`
`VM
`
`BT
`VMM32
`
`Columbia Ex 2043-21
`Symantec v Columbia
`IPR2015-00375
`
`
`
`64-bit Guests And WS 5.5
`(cid:131) Workstation 5.5 enables 64-bit guests
`(cid:131) Currently in beta
`(cid:131) Simultaneously run 32-bit and 64-bit guests
`(cid:131) Runs on 32-bit and 64-bit host OSes
`
`Applications
`
`32-bit or 64-bit
`Host OS
`
`32-
`bit
`VM
`
`64-
`bit
`VM
`
`VMM VMM64
`
`64-bit Hardware
`
`Columbia Ex 2043-22
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Requirements For 64-bit Guests
`(cid:131) Newer hardware required for 64-bit guest
`support
`(cid:131) AMD Opteron Rev. E or later
`(cid:131) AMD Athlon64 Rev. D or later
`(cid:131) Intel VT-enabled processor
`(cid:131) How to determine that you have a 64-bit
`capable system?
`(cid:131) Workstation 5.5 will automatically check to see if
`your CPU meets the requirements
`(cid:131) CPU check utility also available for download on
`WS5.5 beta web page
`(cid:131) http://www.vmware.com/products/beta/ws/
`
`Columbia Ex 2043-23
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Trend No. 3: OS Assist
`(cid:131) Three alternatives for handling non-virtualizable
`instructions
`(cid:131) Binary translation
`(cid:131) Hardware assist
`(cid:131) OS assist or paravirtualization
`
`Compatibility
`Performance
`VMM sophistication
`
`Para-
`virtualization
`
`Binary
`Translation
`Excellent
`Good
`High
`
`Hardware
`Assist
`Excellent
`Average
`Average
`
`Columbia Ex 2043-24
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Paravirtualization
`(cid:131) Paravirtualization can address same problem as
`hardware assist
`(cid:131) Modify the guest OS to remove non-virtualizable instructions
`(cid:131) Export a simpler architecture to OS
`(cid:131) Cannot support unmodified OSes (e.g. Windows 2000/XP)
`(cid:131) Paravirtualization not limited to CPU virtualization
`(cid:131) Higher performance possible
`(cid:131) Relatively easy to add paravirtualization support:
`very difficult to add binary translation
`Hardware
`Binary
`Assist
`Translation
`Excellent
`Excellent
`Average
`Good
`Average
`High
`
`Compatibility
`Performance
`VMM sophistication
`
`Para-
`virtualization
`Poor
`Excellent
`Average
`
`Columbia Ex 2043-25
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Paravirtualization Challenges
`(cid:131) XenoLinux paravirtualization approach
`unsuitable for enterprise use
`(cid:131) Relies on separate kernel for native and in
`virtual machine
`(cid:131) Guest OS and hypervisor tightly coupled
`(cid:131) Tight coupling inhibits compatibility
`(cid:131) Changes to the guest OS are invasive
`(cid:131) Inhibits maintainability and supportability
`(cid:131) Guest kernel must be recompiled when
`hypervisor is updated
`(cid:131) How can we deliver paravirtualization for
`enterprise customers?
`
`Columbia Ex 2043-26
`Symantec v Columbia
`IPR2015-00375
`
`
`
`VMI Paravirtualization API
`(cid:131) VMware proposal: Virtual machine Interface API
`(cid:131) VMI provides maintainability & stability
`(cid:131) API supports low-level and higher-level interfaces
`(cid:131) Allows same kernel to run natively and in a
`paravirtualized virtual machine: “transparent
`paravirtualization”
`(cid:131) Allows for replacement of hypervisors without a guest
`recompile
`(cid:131) Preserve key virtualization functionality: page sharing,
`VMotion, etc.
`(cid:131) We are gathering feedback on the API from many kernel
`developers and OSVs
`(cid:131) http://www.vmware.com/vmi
`(cid:131) http://www.vmware.com/standards/hypercalls.html
`
`Columbia Ex 2043-27
`Symantec v Columbia
`IPR2015-00375
`
`
`
`VMI Paravirtualization
`(cid:131) VMI approach to paravirtualization improves
`compatibility
`(cid:131) API need not compromise performance
`compared to invasive paravirtualization
`
`Compatibility
`Performance
`VMM sophistication
`
`Binary
`Translation
`Excellent
`Good
`High
`
`Hardware
`Assist
`Excellent
`Average
`Average
`
`Para-
`virtualization
`Good
`Excellent
`Average
`
`Columbia Ex 2043-28
`Symantec v Columbia
`IPR2015-00375
`
`
`
`VMware Paravirtualization Performance
`
`Performance Relative to Native
`Bigger is Better
`
`1
`0.9
`0.8
`0.7
`0.6
`0.5
`0.4
`0.3
`0.2
`0.1
`0
`
` compile
`
`Dbench/1client [cpu
`microbenchmark]
`
`Dbench/1client
`HIGHMEM [cpu
`microbenchmark]
`
`Database kernel
`
`Xen2
`
`Xen3-devel
`
`VMware-para
`
`Columbia Ex 2043-29
`Symantec v Columbia
`IPR2015-00375
`
`
`
`VMware And Paravirtualization
`(cid:131) VMware will support paravirtualized
`Linux OSes
`(cid:131) Another guest type when such
`OS’s commercially available
`(cid:131) Flexible architecture
`(cid:131) Use most efficient
`technique for the
`guest OS type
`(cid:131) BT, VT/Pacifica, or
`paravirtualization
`
`VM
`
`VM
`
`VM
`
`VM
`
`Para-
`VMM
`
`VMM32
`
`VMM64
`
`VMM64
`
`Distributed
`VM File
`System
`
`Virtual
`NIC &
`Switch
`
`Storage
`Stack
`
`Network
`Stack
`
`Resource
`Management
`
`Device Drivers
`ESX Server
`
`Hardware
`
`Columbia Ex 2043-30
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Summary
`(cid:131) 64-bit transition happening now
`(cid:131) Virtualization can assist with transition
`(cid:131) 64-bit guests supported in WS5.5
`(cid:131) VMware provides flexible architecture to support
`emerging virtualization technologies
`(cid:131) Multi-mode VMM utilizes binary translation, hardware
`assist and paravirtualization
`(cid:131) Select best operating mode for the workload
`(cid:131) VMware will support paravirtualized guests as they
`appear in enterprise distributions
`(cid:131) VMI offers superior maintainability/flexibility
`(cid:131) Performs as well as invasive paravirtualization
`
`Columbia Ex 2043-31
`Symantec v Columbia
`IPR2015-00375
`
`
`
`PAC346
`VMware and CPU
`Virtualization Technology
`
`Jack Lo
`Sr. Director, R&D
`
`Columbia Ex 2043-32
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Backup slides
`
`Columbia Ex 2043-33
`Symantec v Columbia
`IPR2015-00375
`
`
`
`Performance of Binary Translation
`(cid:131) BT provides many performance optimization
`opportunities
`(cid:131) Fault elimination
`(cid:131) Avoid costs of repeated virtual machine exits
`(cid:131) Binary translator identifies faulting instructions and
`replaces them with special translations
`(cid:131) Jump directly to appropriate handlers without
`an expensive fault
`(cid:131) Guest and VMM share an address space:
`reduces context switch costs
`
`Columbia Ex 2043-34
`Symantec v Columbia
`IPR2015-00375