1111111111111111 IIIIII IIIII 11111 1111111111 111111111111111 11111 1111111111111111 IIII 11111111
`
`US 20050268298Al
`
`(19) United States
`(12) Patent Application Publication
`Hunt et al.
`
`(10) Pub. No.: US 2005/0268298 Al
`Dec. 1, 2005
`( 43) Pub. Date:
`
`(54) SYSTEM, METHOD AND PROGRAM TO
`MIGRATE A VIRTUAL MACHINE
`
`(52) U.S. CI.
`
`.................................................................. 718/1
`
`(75)
`
`Inventors: Jennifer A. Hunt, Endicott, NY (US);
`Steven Shultz, Endicott, NY (US)
`
`(57)
`
`ABSTRACT
`
`Correspondence Address:
`IBM CORPORATION
`IPLAW IQ0N40-3
`1701 NORTH STREET
`ENDICOTT, NY 13760 (US)
`
`(73) Assignee: INTERNATIONAL
`MACHINES
`ARMONK, NY
`
`BUSINESS
`CORPORATION,
`
`(21) Appl. No.:
`
`10/843,931
`
`(22) Filed:
`
`May 11, 2004
`
`Publication Classification
`
`(51)
`
`Int. Cl.7
`
`....................................................... G06F 9/46
`
`A system, method and program product for migrating a first
`virtual machine from a first real computer to a second real
`computer or from a first LPAR to a second LPAR in a same
`real computer. Before migration, the first virtual machine
`comprises an operating system and an application in a first
`private memory private to the first virtual machine. A
`communication queue of the first virtual machine resides in
`a shared memory shared by the first and second computers
`or the first and second LPARs. The operating system and
`application are copied from the first private memory to the
`shared memory. The operating system and application are
`copied from the shared memory to a second private memory
`private to the first virtual machine in the second computer or
`second LPAR. Then, the first virtual machine is resumed in
`the second computer or second LPAR.
`
`read
`
`SHARED MEMORY
`
`50, 90
`
`/
`
`Communication Queue
`:i1
`
`L
`
`I-+
`
`item
`
`[§ "§i
`§ w,o
`
`Block Memory
`
`~
`
`I File I
`
`j Directory J
`
`I Application I ,,.-33
`rl create work 11 Read data
`
`from file
`
`item
`,/
`Virtual Machine
`
`143
`
`7, Linux 0/S
`I WQAF
`I Scheduler I
`
`,.- 83
`
`253
`
`I Application I /34
`~ create work 11 Write data
`
`to file
`
`item
`.,)
`
`144
`
`Virtual Machine
`Linux 0/S
`
`154
`
`,- 84
`
`I Scheduler I
`
`V 7,
`I WQAF
`
`H
`y
`p
`E
`R
`
`I
`s
`0
`R
`
`add work
`
`waiting
`to write
`
`add work
`
`I
`
`~
`
`I Application I ,,,...35
`I create work I r Read data
`
`item
`
`from file
`
`145 /
`
`8
`
`F
`
`Virtual Machine
`CMS 0/S
`
`,.- 165
`
`\ Scheduler 1
`
`acquire
`
`I
`I
`
`File
`
`I
`
`Microsoft Ex. 1013, p. 1
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 1 of 11
`
`US 2005/0268298 Al
`
`FIG 1
`•
`
`/50
`
`Shared Memory
`
`v55
`
`20
`\
`
`25
`
`(
`
`LPAR
`30
`
`p
`H
`y
`s
`I
`C
`A
`L
`
`C
`0
`M
`p
`u
`T
`E
`R
`
`E
`~ so,
`
`l!I Working
`
`Memory
`
`LPAR
`31
`
`C g
`:e
`ro
`a..
`"iii
`(J
`·ci
`0
`...J
`
`Virtual VPM tf3
`H
`y Machine App
`, - - 43
`33
`p
`0/S V
`E Virtual VPM
`, - -1.)54
`R
`Machine App
`V 34
`r - - 44
`0/S v
`I
`Machine App - Communication
`s Virtual VPM
`-
`0
`R
`0/S':;;l_
`
`35
`
`r - -
`
`40
`'45
`I - - - Storage I'\
`I
`26
`
`Queue 52
`
`Block
`Memory54
`
`Virtual VPM
`Machine r - -
`App
`33M , - -
`0/S
`Virtual VPM
`-
`Machine App
`H
`34M c - - -
`y
`51
`0/S
`Shared Memory L/
`p
`Virtual VPM
`E
`, - -~ Communication
`Machine App
`R
`-
`35M
`A Queue 52-1
`-
`V
`I Virtual VPM
`Block
`s Machine c - - -l\45M Memory 54-1
`App
`0 36
`,___ /"156
`0/S i---46
`R
`Virtual VPM
`-
`57
`Machine App l/
`37 ~
`0/S -r;,
`
`0/S rt -
`
`~
`
`CPU
`23
`. - -
`RAM
`24
`
`81.....,__
`Working
`Memory 41
`r·: :··: r-:
`'
`
`7
`
`-11 Console i 14---27
`
`I - \
`
`1
`
`110
`
`Microsoft Ex. 1013, p. 2
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 2 of 11
`.
`FIG 2
`
`US 2005/0268298 Al
`
`_,I
`
`I
`
`,..-so
`
`./
`,---
`
`Shared Memory
`
`Communication
`
`Block
`Memory54
`
`Virtual VPM 53
`-
`H
`y Machine App V
`...--- 43
`33
`120
`p
`0/S
`Physical
`E Virtual VPM
`-
`54
`Computer R
`V 34 - 44
`Machine App ~
`0/S V
`s Virtual VPM 55
`~23
`u
`-
`124 0
`-
`Machine App
`R
`RAM
`35
`00 ~ Queue 52
`80-....,_
`Working 40
`'45
`Memory
`t - - Storage f\
`I
`26
`
`Virtual VPM -
`
`Machine App
`33M ~
`0/S
`Virtual VPM
`-
`Machine
`App
`H
`...---
`34M
`51
`y
`Shared Memory v
`0/S
`- ~ Communication
`p
`Virtual VPM
`E
`Machine App
`-
`0/S rr_
`-
`R
`35M
`V
`- l\45M Memory 54-1 -
`I Virtual VPM
`s
`Machine App
`0
`I"'\
`...--- 56
`36
`0/S r---46
`R
`Virtual VPM i.31
`-
`37 -0/S ~f),7
`Machine App
`
`121
`Physical
`Computer
`
`~25
`u
`126
`RAM
`81-....,_
`Working
`Memory 41
`;·: :··; :-·:
`
`,;; A Queue
`
`_
`1
`
`52
`
`~
`
`Block
`
`~ I[ Console ij ~ 2 7
`
`I -
`
`\
`
`l
`
`210
`
`Microsoft Ex. 1013, p. 3
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 3 of 11
`
`US 2005/0268298 Al
`
`FIG. 3
`
`,310
`
`Shared
`Storage
`§Q
`
`Shared Memory
`50
`
`Shared Memory
`90
`
`Virtual VPM
`. -
`Machine App -
`33
`, - -
`0/S
`Virtual VPM
`.
`i - - -
`Mach in e App
`34
`i - - -
`~~
`~--~O_I_S~
`Virtual VPM 55
`M Machine
`Machine App ~"'------r--+-----~ Private
`35
`resume
`Memory
`-
`0/S
`Contents
`
`H
`y
`p
`E
`R
`V
`I
`s
`0
`R
`
`40
`
`'45
`
`20
`\
`
`25
`
`(
`
`LPAR
`30
`
`p
`H
`y
`s
`I
`C
`A
`L
`
`C
`0
`M
`p
`u
`T
`E
`R
`
`E
`....
`(ll
`C>
`0 ....
`Q.
`(/)
`C
`
`0 :e
`t::
`(ll
`Q.
`ro
`0
`'oi
`0
`...J
`
`r - - -
`CPU
`23
`r - - -
`RAM
`24
`
`'-35-A
`
`Shared Memory
`51
`
`attach
`
`attach
`
`,;-52
`Communication
`Queue
`154
`Block
`Memory
`
`1
`1
`
`-
`-
`
`LPAR
`31
`
`H
`y
`p
`E
`R
`V
`I
`s
`0
`R
`
`41
`
`1 - - -
`
`Virtual VPM
`Machine App
`36
`, - -
`0/S
`Virtual VPM
`1 - - -
`Machine App
`,___
`37
`0/S
`Virtual VPM
`Machine App
`,___ -
`35M
`0/S -
`45M
`
`--
`
`M
`
`-
`
`Shared
`Storage
`61
`
`-B-27
`
`I
`
`-
`
`\
`
`Microsoft Ex. 1013, p. 4
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 4 of 11
`
`US 2005/0268298 Al
`
`FIG 4
`
`•
`
`410
`/
`
`Shared Memory
`50
`
`Shared Memory
`90
`
`Virtual VPM
`,___
`Shared
`H
`y Machine App ,_ Storage
`120
`60
`p
`0/S
`Physical
`E Virtual VPM
`Computer R
`Machine App
`-
`V
`34
`0/S
`s Virtual VPM
`-
`0
`Machine App
`R
`0/S
`
`t - - - -
`
`resume
`
`Machine App
`0/S
`Virtual VPM
`r - - -
`Machine App
`0/S
`Virtual VPM
`,___
`Machine App
`0/S -
`
`-
`
`Shared Memory
`51
`
`M
`
`attach
`
`attach
`
`Virtual
`M Machine
`Private
`-
`Memory
`Contents
`
`'--35-A
`
`I
`
`/52
`;communication
`Queue
`,/54
`Block
`Memory_
`
`33 -
`~ I
`35 -
`Virtual VPM -
`36 -
`37 -
`~ R 35M - -
`
`~
`
`;
`
`u
`
`4
`
`M
`
`40
`
`121
`Physical
`Computer
`
`H
`y
`p
`E
`
`u
`
`M
`
`V
`I
`6 s
`0
`R
`
`Shared
`Storage
`61
`
`41
`
`..._ II Console ~
`
`I --=- \,.
`
`27
`
`Microsoft Ex. 1013, p. 5
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 5 of 11
`
`US 2005/0268298 Al
`
`20
`\
`
`25
`
`(
`
`LPAR
`30
`
`LPAR
`31
`
`p
`H
`y
`s
`I
`C E
`...
`n:I
`A
`O'l
`e
`a..
`L
`(/J
`C:
`0
`:;::
`t
`n:I
`a..
`cii
`-~
`
`O'l
`0
`....J
`
`C
`0
`M
`p
`u
`T
`E
`R
`
`-
`
`CPU
`23
`, - .
`RAM
`24
`
`40
`
`41
`
`Virtual VPM
`,..._
`H
`y Machine App
`,__
`33
`p
`0/S
`E
`Virtual VPM
`R ~
`Machine App
`V
`34 ~
`I
`0/S
`s
`Virtual VPM
`,..._
`0
`Machine App .---
`,__
`R
`35
`0/S
`
`.
`FIG 5
`
`,-510
`
`Shared Storage 560
`Virtual
`M Machine M
`Private
`Memory
`Contents
`35-2
`
`~I Communication ~
`I Queue 52-2
`
`Ml Block r
`
`-I Memory 54-2
`
`/""so
`
`Shared Memory
`
`-
`
`Communication
`Queue 52
`
`......
`
`Block
`Memory 54
`
`-
`
`Virtual VPM
`,__
`Machine App
`33M
`r - - -
`0/S
`Virtual VPM
`,..._
`Machine App
`H
`,__
`34M
`y
`0/S
`p
`Virtual VPM
`r - - - - R
`E
`Machine App t+-A
`R
`35M
`V
`I Virtual VPM
`s Machine r - - - -
`App
`0 36
`r - - -
`0/S
`R
`Virtual VPM
`, - -
`Machine
`App
`,__
`37
`0/S
`
`r - - -
`
`0/S rt
`
`Shared Memory v
`
`51
`
`Communication
`Queue 52_3
`
`Block
`Memory 54-3
`
`-
`
`-11 Console ~ i-----27
`
`L"'="'\
`
`Microsoft Ex. 1013, p. 6
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 6 of 11
`
`US 2005/0268298 Al
`
`FIG 6
`
`•
`
`,,,,so
`
`Shared Memory
`
`Communication
`Queue 52
`
`610
`/
`Shared Storage 560
`Virtual
`Machine M
`Private
`Memory
`Contents
`35-2
`
`M
`
`M
`
`Communication M
`Queue 52-2
`
`1 - -
`
`L......._
`
`Block
`Memory 54
`
`M
`
`M
`Block
`Memory 54-2 I-
`
`120
`Physical
`Computer
`
`u
`
`M
`
`-
`VPM
`Virtual
`H
`y Machine App
`-
`33
`p
`0/S
`-
`E
`Virtual VPM
`R
`Machine App
`V
`-
`34
`0/S
`-
`s Virtual VPM
`4 0
`Machine App
`,...._
`R
`,___
`35
`0/S -
`
`40
`
`~ I
`Virtual VPM -
`
`Machine App
`,____
`36
`0/S
`Virtual VPM
`.....__
`Machine App
`.....__
`37
`0/S
`Virtual VPM
`,___ R
`Machine App -A
`,___
`
`121
`Physical
`Computer
`
`H
`y
`p
`E
`
`~ R 35M
`0/S rt
`-
`0 36 -
`-
`37 -0/S
`
`u
`
`6
`
`M
`
`V
`I Virtual VPM
`s
`Machine App
`0/S
`Virtual VPM
`Machine App
`
`R
`
`41
`
`51
`Shared Memory l../
`
`Communication
`Queue
`_
`3
`
`52
`
`Block
`Memory 54-3
`
`- II Console ~ i---27
`
`I ----=--
`\
`-'LID\
`
`Microsoft Ex. 1013, p. 7
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 7 of 11
`
`US 2005/0268298 Al
`
`FIG. 7
`I Application I r33
`rl create work 11 Read data I
`I
`7, Linux 0/S
`I WQAF
`I Scheduler I
`
`from file
`
`item
`./
`Virtual Machine
`
`143
`
`read
`
`so, 90
`
`/
`
`SHARED MEMORY
`
`Communication Queue
`52
`
`253
`
`/' 83
`
`add work § §
`I
`
`k
`m
`
`k
`m
`
`Work
`item
`
`/12
`
`-
`
`I Application I /34
`p 1 create work 11 Write data
`I l,
`I WQAF
`
`item
`./
`
`to file
`
`144
`
`H
`y
`
`E
`R
`V
`
`s
`0
`R
`
`Virtual Machine
`Linux 0/S
`
`154
`
`,.-84
`
`I Scheduler I
`
`waiting
`to write
`
`add work
`
`Block Memory
`54
`
`I File I
`I Directory J
`
`40
`
`I Application I /
`I create work 11 Read data
`
`from file
`
`item
`
`35
`
`145 /
`
`Virtual Machine
`CMS 0/S
`
`155
`
`I W~AF I
`
`/" 165
`
`I Scheduler
`
`acquire
`
`I
`File
`
`I
`I
`
`I
`
`Microsoft Ex. 1013, p. 8
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 8 of 11
`
`US 2005/0268298 Al
`
`item
`./
`
`143
`
`from file
`
`Virtual Machine
`Linux 0/S
`
`/' 50, 90
`
`SHARED MEMORY
`
`Communication Queue
`52
`
`k
`m
`
`k
`m
`
`Work
`item
`
`/" 72
`
`-
`
`FIG. 8
`I Application I r33
`~ create work 11 Read data I
`71
`add work § §
`I
`I WQAF
`
`73
`
`/' 83
`
`Scheduler I
`
`I Application I /34
`
`H
`y
`p ~ create work 11 Write data
`item
`to file
`E
`./
`R
`V
`
`144
`
`I 11
`I WQAF
`
`s
`0
`R
`
`Virtual Machine
`Linux 0/S
`
`74
`
`~84
`
`I Scheduler I
`
`~
`
`Application
`
`I
`I / 3 5
`I cre~te work I Read data
`
`item
`
`from file
`
`add work
`
`Block Memory
`54
`
`I File I
`I Directory I
`
`read
`
`145 /
`Virtual Machine
`CMS 0/S
`75
`
`I W~AF I
`
`Write data
`to file
`
`/' 85
`
`I Scheduler -
`
`acquire
`
`I
`
`I
`I
`
`File
`
`Microsoft Ex. 1013, p. 9
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 9 of 11
`
`US 2005/0268298 Al
`
`FIG. 9
`
`INITIATE MIGRATION
`t
`STOP VIRTUAL MACHINES IN ORIGINAL HYPERVISOR
`f
`COMMUNICATE VPM CONTENTS AND VIRTUAL ADDRESSES OF COMMUNICATION
`903
`QUEUE AND BLOCK MEMORY TO DESTINATION HYPERVISOR
`t
`COMMUNICATE COMMUNICATIONS QUEUE TO DESTINATION HYPERVISOR 904
`
`9021
`
`COMMUNICATE BLOCK MEMORY TO DESTINATION HYPERVISOR
`
`DELETE VIRTUAL MACHINE FROM ORIGINAL HYPERVISOR
`
`ALLOCATE RESOURCES TO MIGRATED VIRTUAL MACHINE IN
`DESTINATION HYPERVISOR
`
`'
`'
`'
`•
`
`908
`
`9121
`
`928
`
`930
`
`934
`
`940
`
`944
`
`950
`
`954
`
`994
`
`TRANSFER VPM CONTENTS AND VIRTUAL ADDRESSES OF COMMUNICATION QUEUE
`AND BLOCK MEMORY TO PRIVATE MEMORY OF MIGRATED VIRTUAL MACHINE
`IN DESTINATION HYPERVISOR
`
`•
`'
`
`RESUME MIGRATED VIRTUAL MACHINE
`
`TRANSFER COMMUNICATION QUEUE TO SHARED MEMORY OF
`DESTINATION HYPERVISOR
`t
`TRANSFER BLOCK MEMORY TO SHARED MEMORY OF
`DESTINATION HYPERVISOR
`+
`ATTACH COMMUNICATION QUEUE TO MIGRATED VIRTUAL MACHINE
`t
`ATTACH BLOCK MEMORY TO MIGRATED VIRTUAL MACHINE
`
`NO
`
`/998
`:wA1TI
`
`990
`
`WORK
`ITEMS ON QUEUE
`?
`YES
`PROCESS WORK ITEM
`
`Microsoft Ex. 1013, p. 10
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 10 of 11
`
`US 2005/0268298 Al
`
`FIG. 10
`
`INITIATE MIGRATION
`
`COPY VPM CONTENTS AND VIRTUAL ADDRESSES OF COMMUNICATION QUEUE
`1000
`AND BLOCK MEMORY TO SHARED MEMORY
`
`STOP VIRTUAL MACHINE BEING MIGRATED
`
`1002
`
`DELETE VIRTUAL MACHINE FROM ORIGINAL HYPERVISOR
`
`1012
`
`ALLOCATE RESOURES TO MIGRATED VIRTUAL MACHINE
`
`1028
`
`TRANSFER VPM CONTENTS AND VIRTUAL ADDRESSES OF COMMUNICATION QUEUE
`AND BLOCK MEMORY TO PRIVATE MEMORY OF MIGRATED VIRTUAL MACHINE
`1030
`IN DESTINATION HYPERVISOR
`
`RESUME MIGRATED VIRTUAL MACHINE
`
`1034
`
`ATTACH COMMUNICATION QUEUE TO MIGRATED VIRTUAL MACHINE
`
`1050
`
`ATTACH BLOCK MEMORY TO MIGRATED VIRTUAL MACHINE
`
`1054
`
`PROCESS WORK ITEM
`
`1094
`
`Microsoft Ex. 1013, p. 11
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`Patent Application Publication Dec. 1, 2005 Sheet 11 of 11
`
`US 2005/0268298 Al
`
`FIG. 11
`
`INITIATE MIGRATION
`t
`COPY VPM CONTENTS AND VIRTUAL ADDRESSES OF COMMUNICATION QUEUE
`1100
`AND BLOCK MEMORY TO SHARED STORAGE
`+
`STOP VIRTUAL MACHINE BEING MIGRATED
`♦
`COMMUNICATE COMMUNICATIONS QUEUE TO SHARED STORAGE
`♦
`COMMUNICATE BLOCK MEMORY TO SHARED STORAGE
`
`1102
`
`1104
`
`1108
`
`'
`
`DELETE VIRTUAL MACHINE FROM ORIGINAL HYPERVISOR
`♦
`ALLOCATE RESOURCES TO MIGRATED VIRTUAL MACHINE
`♦
`TRANSFER VPM CONTENTS AND VIRTUAL ADDRESSES OF COMMUNICATION QUEUE
`AND BLOCK MEMORY TO PRIVATE MEMORY OF
`DESTINATION HYPERVISOR
`
`1112
`
`1128
`
`1130
`
`11341
`
`1140
`
`1144
`
`1150
`
`1154
`
`11941
`
`'
`
`RESUME MIGRATED VIRTUAL MACHINE
`♦
`TRANSFER COMMUNICATION QUEUE TO SHARED MEMORY OF
`DESTINATION HYPERVISOR
`♦
`TRANSFER BLOCK MEMORY TO SHARED MEMORY OF
`DESTINATION HYPERVISOR
`
`'
`
`ATTACH COMMUNICATION QUEUE TO MIGRATED VIRTUAL MACHINE
`♦
`ATTACH BLOCK MEMORY TO MIGRATED VIRTUAL MACHINE
`
`NO
`
`/1198
`:wA1TI
`
`1190
`
`WORK
`ITEMS ON QUEUE
`?
`YES
`PROCESS WORK ITEM
`
`Microsoft Ex. 1013, p. 12
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`US 2005/0268298 Al
`
`Dec. 1, 2005
`
`1
`
`SYSTEM, METHOD AND PROGRAM TO
`MIGRATE A VIRTUAL MACHINE
`
`BACKGROUND OF THE INVENTION
`
`[0001] The invention relates generally to computer sys(cid:173)
`tems, and deals more particularly with migration of a virtual
`machine within a same real computer from one logical
`partition to another, or from one real computer to another
`real computer.
`
`[0002] A virtual machine operating system is well known
`today, and includes a hypervisor program, and separate
`virtual machines formed by the hypervisor. In an IBM zNM
`operating system, the hypervisor program is called the
`Control Program ("CP"). Each virtual machine is also called
`a "user portion" or "guest". A virtual machine is a virtual
`sharing/partitioning of computer resources such as proces(cid:173)
`sor(s ), memory, storage and 1/0 (i.e. network cards, printers
`and displays.) A guest operating system executes/runs on
`each virtual machine. One or more applications run on each
`guest operating system.
`
`[0003]
`It was also known to logically partition a computer
`by logically dividing the real computer resources. The user
`defined each logical partition ("LPAR"), i.e. the amount of
`processors, memory and storage for each LPAR. Each LPAR
`could be allocated specific real computer resources or a
`share of the total computer resources. Then, in some com(cid:173)
`puters, a separate hypervisor was loaded into each LPAR to
`form multiple virtual machines in each logical partition.
`Each such virtual machine was a virtual sharing of the
`resources allocated to its LPAR.
`[0004] Even though each application and guest operating
`system are executing in a virtual machine, they operate as if
`they are running on their own private, real computer. The
`following is an example of how a known virtual machine
`utilizes its processor or share of processor time to perform
`work items. Each virtual machine has its own synchroniza(cid:173)
`tion or lock function, work queue assignment function, work
`scheduler and associated queue of work items or tasks
`assigned to the virtual machine. The synchronization or lock
`function, work queue assignment function, work scheduler
`and the work queue are all private to the virtual machine in
`this example. The synchronization or lock function manages
`locks for a work queue to control which work items must run
`sequentially and which tasks can run in parallel. The work
`queue assignment function is a program function within the
`virtual machine which adds work items to the work queue of
`the virtual machine when generated by the virtual machine.
`The work items are added to the queue at a position based
`on an assignment algorithm. The assignment algorithm may
`consider such factors as relative priority level of each work
`item and the order in which work items were created, i.e.
`first in first out. Each work item on the queue includes
`information indicating its type, and therefore, which func(cid:173)
`tion within the virtual machine is best suited to handle it. A
`"work scheduler" is a program function which schedules
`each of the work items on its queue for execution. The work
`scheduler passes the work items to the appropriate function
`within the virtual machine for execution by the virtual
`processor.
`
`[0005]
`It was also known for multiple virtual machines to
`share a work queue to distribute the work items amongst the
`virtual machines and their respective shares of real proces-
`
`sors. A server virtual machine was utilized for the purpose
`of "hosting" this shared work queue for the other, "working"
`virtual machines. The shared work queue resides in memory
`private to the server virtual machine. When a working
`virtual machine creates a new work item, and the work
`queue assignment function for this working virtual machine
`decides to send this new work item to the server virtual
`machine, it uses a communication protocol (e.g. TCP/IP)
`and a virtual 1/0 device driver to send that work item to this
`server virtual machine. Then, the server virtual machine
`places the new work item on the shared work queue in an
`order determined by the server virtual machine. When the
`virtual CPU within a working virtual machine is available to
`execute a work item on the shared work queue, the work
`scheduler within this working virtual machine uses a com(cid:173)
`munication protocol and virtual 1/0 device driver to make
`that request to the server virtual machine. In response, the
`server virtual machine uses a communication protocol to
`send a work item to the working virtual machine that made
`the request. While this arrangement provides a shared work
`queue, it requires a high overhead communication protocol
`to both send a work item to the work queue and obtain a
`work item from the work queue.
`
`[0006] U.S. patent application entitled "Management of
`Virtual Machines to Utilize Shared Resources" Ser. No.
`10/425,470, filed Apr. 29, 2003 by Casey et al., discloses the
`"cloning" of a virtual machine, including its operating
`system and application(s), when
`the application(s)
`is
`resource constrained. This will increase the proportion of
`total computer resources allocated to the application(s)
`because there is an additional virtual machine (with its share
`of resources) running the application(s). This patent appli(cid:173)
`cation is hereby incorporated by reference as part of the
`present disclosure. U.S. patent application entitled "Man(cid:173)
`agement of Locks in a Virtual Machine Environment" Ser.
`No. 10/425,468, filed Apr. 29, 2003 by Donovan et al.
`discloses a shared memory with a work queue and work
`queue lock structure shared by multiple virtual machines.
`The multiple virtual machines can directly access the shared
`lock structure and shared work queue. This patent applica(cid:173)
`tion is hereby incorporated by reference as part of the
`present disclosure.
`
`[0007]
`It was known for a computer to include a physical
`communication card that was inserted into the computer.
`When the communication card receives a message from
`another computer, the communication card sends an inter(cid:173)
`rupt to a CPU within the computer. In response, the CPU
`will invoke a program function within the computer to fetch
`and handle the message. The physical communication card
`could be removed and inserted into another computer. Any
`messages contained in memory within the physical commu(cid:173)
`nication card and not yet read by the original computer
`would not be available to this other personal computer. Also,
`messages sent to the physical communication card during its
`movement from the original computer to the other personal
`computer would be lost.
`
`[0008]
`It was also known for a computer to include a
`physical block 1/0 card to write data to and read data from
`(disk) storage. During a write mode, the CPU of the com(cid:173)
`puter passes a block of data to the block 1/0, and requests
`that it be written to storage. In response, the block 1/0 card
`writes the data to the storage, and then sends an interrupt
`back to the CPU indicating that the 1/0 completed. When
`
`Microsoft Ex. 1013, p. 13
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`US 2005/0268298 Al
`
`Dec. 1, 2005
`
`2
`
`receiving the interrupt, the CPU knows that the block of data
`was successfully written to storage, and then can proceed
`accordingly, for example, erasing the data from memory.
`During a read mode, the CPU requests the block 1/0 card to
`read a specified block of data from storage. In response, the
`block 1/0 card reads the data from storage and writes it to a
`buffer accessible to the CPU. Then, the block 1/0 card sends
`an interrupt back to the CPU indicating that the 1/0 com(cid:173)
`pleted. After receiving the interrupt, the CPU can read the
`data from the buffer. The physical block 1/0 card could be
`removed and inserted into another computer. However, any
`1/0 requests currently in progress on the physical block 1/0
`card during its movement from the original computer to the
`other personal computer would be lost.
`
`[0009]
`It was known to migrate a virtual machine from
`one real computer to another real computer and within one
`real computer from one LPAR to another LPAR. Adesse
`Corporation's Single System Image could save the state of
`a virtual machine and migrate that virtual machine, but only
`if there was no 1/0 in progress and the virtual machine had
`no communication devices. A research project entitled
`"Guest Save/Restore Facility" by Brookhaven National
`Laboratory's could save the state of a virtual machine and
`resume that virtual machine at some future time, but only if
`there was no 1/0 in progress and the virtual machine had no
`Inc.'s Distributed
`communication devices. MiraSoft,
`devices could save the state of a virtual machine and migrate
`that virtual machine, but only if there was no 1/0 in progress
`and the virtual machine had no communication devices.
`With these three products, no inter-virtual machine commu(cid:173)
`nication was permitted. There was no ability to handle "in
`flight" 1/0, i.e. communications and data sent from one
`virtual machine to another virtual machine but not yet
`received or processed by the other virtual machine.
`
`[0010] VMWare Corporation's VMMotion program
`migrates an application, including its program code, state
`information, registers, memory, etc., from one real computer
`to another real computer. The computer system in which the
`application executes uses a communication device which
`comprised a virtual network interface card. Before the
`migration of the application, incoming communications
`were stopped for some period and prior communications
`were completed so there would be no "in flight" communi(cid:173)
`cations during the migration. The computer system in which
`the application executes also uses a disk driver and a disk for
`storage of block data. Before the migration of the applica(cid:173)
`tion, disk 1/0 operations were stopped for some period and
`prior disk 1/0 operations were completed so there would be
`no unaccounted 1/0 during the migration.
`
`[0011] There is currently an Open Source project named
`"Partition image" directed to moving a Linux image from
`one real computer to another real computer. It saves the state
`of the Linux image to disk and this image can then be
`migrated to another computer. However, all communication
`and disk 1/0 must be completed and stopped before the
`image is saved. Also, a Tivoli System Automation program
`moves applications from one computer to another computer.
`The computer system in which the application executes uses
`a physical card for communication from the source com(cid:173)
`puter to the target computer. The computer system uses a
`device driver and disk for storage of blocks of data. Before
`migration, the communication device is stopped for some
`period and the prior communications completed, so there
`
`would be no in flight communications during the migration.
`Likewise, disk 1/0 operations are stopped for some period
`and prior 1/0 requests completed before migration, so there
`would be no unaccounted 1/0 during the migration.
`
`[0012] An object of the present invention is to efficiently
`migrate a virtual machine within a same real computer from
`one logical partition to another, or from one real computer
`to another real computer.
`
`[0013] Another object of the present invention is to
`migrate a virtual machine while communications to the
`virtual machine are in progress, without losing the commu(cid:173)
`nications or stopping subsequent communications for an
`appreciable amount of time.
`
`SUMMARY OF THE INVENTION
`
`[0014] The invention resides in a system, computer pro(cid:173)
`gram product and method for migrating a first virtual
`machine and a communication queue from a first real
`computer to a second real computer or from a first LPAR to
`a second LPAR in a same real computer. Before migration,
`the first virtual machine comprises an operating system and
`an application in a first private memory private to the first
`virtual machine. Before migration, the communication
`queue resides in a first shared memory shared by the first
`virtual machine and a second virtual machine in the first
`computer or the first LPAR. The first virtual machine is
`stopped in the first computer or the first LPAR. The first
`computer or the first LPAR communicates the operating
`system and application to the second computer or the second
`LPAR, and the second computer or the second LPAR writes
`the operating system and application into a second private
`memory private to the first virtual machine in the second
`computer or the second LPAR. The first computer or the first
`LPAR communicates the communication queue to the sec(cid:173)
`ond computer or second LPAR, and the second computer or
`the second LPAR writes the communication queue into a
`second shared memory shared by the second and third and
`fourth virtual machines in the second computer or the
`second LPAR. The first virtual machine is resumed in the
`second computer or second LPAR.
`
`[0015] According to another feature of the present inven(cid:173)
`tion, there is a system, method and program product for
`migrating a first virtual machine from a first real computer
`to a second real computer or from a first LPAR to a second
`LPAR in a same real computer. Before the migration, the
`first virtual machine comprises an operating system and an
`application in a first private memory private to the first
`virtual machine. A communication queue resides in a first
`shared memory shared by the first virtual machine and a
`second virtual machine in the first computer or the first
`LPAR. The operating system and application are copied
`from the first private memory to a shared storage shared by
`the first and second computers or the first and second
`LPARs. The communication queue is copied from the first
`shared memory to the shared storage. The second real
`computer or second LPAR is notified of the operating
`system, application and communication queue in the shared
`storage. The operating system and application are copied
`from the shared storage to a second private memory private
`to the first virtual machine in the second computer or second
`LPAR. The first virtual machine is resumed in the second
`computer or second LPAR. The communication queue is
`
`Microsoft Ex. 1013, p. 14
`Microsoft v. Daedalus Blue
`IPR2021-00832
`
`

`

`US 2005/0268298 Al
`
`Dec. 1, 2005
`
`3
`
`copied from the shared storage to a second shared memory
`shared by the first, third and fourth virtual machines in the
`second computer or second LPAR.
`
`[0016] According to another feature of the present inven(cid:173)
`tion, there is a system, method and program product for
`migrating a first virtual machine from a first real computer
`to a second real computer or from a first LPAR to a second
`LPAR in a same real computer. Before migration, the first
`virtual machine comprises an operating system and an
`application in a first private memory private to the first
`virtual machine. A communication queue of the first virtual
`machine resides in a shared memory shared by the first and
`second computers or the first and second LPARs. The
`operating system and application are copied from the first
`private memory to the shared memory. The operating system
`and application are copied from the shared memory to a
`second private memory private to the first virtual machine in
`the second computer or second LPAR. Then, the first virtual
`machine is resumed in the second computer or second
`LPAR.
`
`BRIEF DESCRIPTION OF THE FIGURES
`
`[0017] FIG. 1 is a block diagram of a real computer with
`two logical partitions, respective shared memories and mul(cid:173)
`tiple virtual machines in each logical partition, and illus(cid:173)
`trates a process according to one embodiment of the present
`invention for migrating a virtual machine from one logical
`partition to the other logical partition.
`
`[0018] FIG. 2 is a block diagram of two separate, real
`computers, respective shared memories and multiple virtual
`machines in each real computer, and illustrates a process
`according to another embodiment of the present invention
`for migrating a virtual machine from one real computer to
`the other real computer.
`
`[0019] FIG. 3 is a block diagram of a real computer with
`two logical partitions, multiple virtual machines in each
`logical partition, and a memory shared by both logical
`partitions, and illustrates a process according to another
`embodiment of the present invention for migrating a virtual
`machine from one logical partition to the other logical
`partition.
`
`[0020] FIG. 4 is a block diagram of two separate, real
`computers, multiple virtual machines in each real computer,
`and a memory shared by both real computers, and illustrates
`a process according to another embodiment of the present
`invention for migrating a virtual machine from one real
`computer to the other real computer.
`
`[0021] FIG. 5 is a block diagram of a real computer with
`two logical partitions, respective shared memories, storage
`shared by both logical partitions and multiple virtual
`machines in each logical partition, and illustrates a process
`according to one embodiment of the present invention for
`migrating a virtual machine from one logical partition to the
`other logical partition.
`
`[0022] FIG. 6 is a block diagram of two separate, real
`computers, respective shared memories, storage shared by
`both computers, and multiple virtual machines in each real
`computer, and illustrates a process according to another
`embodiment of the present invention for migrating a virtual
`machine from one real computer to the other real computer.
`
`[0023] FIG. 7 is a block diagram illustrating a communi(cid:173)
`cation queue for a virtual machine and a shared block
`memory, within the shared memories of FIGS. 1, 2, 3, 4, 5
`and 6.
`
`[0024] FIG. 8 is a block diagram illustrating a communi(cid:173)
`cation queue and a dedicated block memory for a virtual
`machine, within the shared memories of FIGS. 1, 2, 3, 4, 5
`and 6.
`
`[0025] FIG. 9 is a flow chart illustrating the virtual
`machine migration process within the computer systems of
`FIGS. 1 and 2.
`
`[0026] FIG. 10 is a flow chart illustrating the virtual
`machine migration process within the computer systems of
`FIGS. 3 and 4.
`[0027] FIG. 11 is a flow chart illustrating the virtual
`machine migration process within the computer systems of
`FIGS. 5 and 6.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`
`[0028] The present invention will now be described in
`detail with reference to the figures, wherein like reference
`numbers indicate like elements throughout. FIG. 1 illus(cid:173)
`trates a computer system generally designated 110 in accor(cid:173)
`dance with one embodiment of the present invention. Com(cid:173)
`puter system 110 includes a real/physical computer 20 which
`includes a CPU 23, RAM 24, logical partitioning program
`25, network ports (not shown) and operator console 27.
`Compu