`
`
`
`I IIIII IIIIIIII 1111111111111fORIg111111 111111 IIII
`
`I 1111111111111111 11111 111111111111111 11111 111111111111111 111111111111111111
`
`
`
`
`US009875160B2
`
`c12) United States Patent
`(12) United States Patent
`Bezbaruah et al.
`Bezbaruah et al.
`
`(IO) Patent No.:
`(10) Patent No.:
`(45) Date of Patent:
`(45) Date of Patent:
`
`US 9,875,160 B2
`US 9,875,160 B2
`*Jan. 23, 2018
`*Jan. 23, 2018
`
`(54) EFFICIENTLY PROVIDING VIRTUAL
`EFFICIENTLY PROVIDING VIRTUAL
`MACHINE REFERENCE POINTS
`MACHINE REFERENCE POINTS
`
`(71) Applicant: MICROSOFT TECHNOLOGY
`Applicant: MICROSOFT TECHNOLOGY
`LICENSING, LLC, Redmond, WA
`LICENSING, LLC, Redmond, WA
`(US)
`(US)
`
`(72)
`
`Inventors: Angshuman Bezbaruah, Redmond,
`Inventors: Angshuman Bezbaruah, Redmond,
`WA (US); Lars Reuther, Kirkland, WA
`WA (US); Lars Reuther, Kirkland, WA
`(US); Taylor O'Neil Brown, Bellevue,
`(US); Taylor O'Neil Brown, Bellevue,
`WA (US); John Andrew Starks,
`WA (US); John Andrew Starks,
`Seattle, WA (US)
`Seattle, WA (US)
`
`(73) Assignee: Microsoft Technology Licensing, LLC,
`Assignee: Microsoft Technology Licensing, LLC,
`Redmond, WA (US)
`Redmond, WA (US)
`
`( *) Notice:
`Notice:
`
`Subject to any disclaimer, the term ofthis
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`U.S.C. 154(b) by 0 days.
`
`This patent is subject to a terminal dis(cid:173)
`This patent is subject to a terminal dis-
`claimer.
`claimer.
`
`(21) Appl. No.: 15/219,958
`(21) Appl. No.: 15/219,958
`
`(22) Filed:
`(22) Filed:
`
`Jul. 26, 2016
`Jul. 26, 2016
`
`(65)
`(65)
`
`Prior Publication Data
`Prior Publication Data
`
`(58) Field of Classification Search
`(58) Field of Classification Search
`CPC ........ G06F 3/0619; G06F 3/065; G06F 3/067;
`CPC
`G06F 3/0619; G06F 3/065; G06F 3/067;
`G06F 9/45558; G06F 11/1451; G06F
`G06F 9/45558; G06F 11/1451; G06F
`2201/84
`2201/84
`
`(Continued)
`(Continued)
`
`(56)
`(56)
`
`References Cited
`References Cited
`
`U.S. PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`
`5,721,915 A
`5,721,915 A
`6,411,964 B1
`6,411,964 Bl
`
`2/1998 Sockut et al.
`2/ 1998 Sockut et al.
`6/2002 Iyer et al.
`6/2002 Iyer et al.
`(Continued)
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`FOREIGN PATENT DOCUMENTS
`
`CN
`CN
`
`103946807 A
`103946807 A
`
`7/2014
`7/2014
`
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`
`Zhang et al., "Optimizing VM Checkpointing for Restore Perfor-
`Zhang et al., "Optimizing VM Checkpointing for Restore Perfor(cid:173)
`mance in VMware ESXi", 2013, USENIX Annual Technical Con-
`mance in VMware ESXi", 2013, USENIX Annual Technical Con(cid:173)
`ference, 12 pages.*
`ference, 12 pages.*
`
`(Continued)
`(Continued)
`
`Primary Examiner - Ted T Vo
`Primary Examiner — Ted T Vo
`(74) Attorney, Agent, or Firm - Workman Nydegger
`(74) Attorney, Agent, or Firm — Workman Nydegger
`
`US 2017/0109240 Al
`US 2017/0109240 Al
`
`Apr. 20, 2017
`Apr. 20, 2017
`
`(57)
`(57)
`
`ABSTRACT
`ABSTRACT
`
`Related U.S. Application Data
`Related U.S. Application Data
`
`(63) Continuation of application No. 14/573,976, filed on
`(63) Continuation of application No. 14/573,976, filed on
`Dec. 17, 2014, now Pat. No. 9,430,272.
`Dec. 17, 2014, now Pat. No. 9,430,272.
`
`Int. Cl.
`(51)
`(51) Int. Cl.
`G06F 9/455
`G06F 9/455
`G06F 11114
`G06F 11/14
`G06F 3/06
`G06F 3/06
`(52) U.S. Cl.
`(52) U.S. Cl.
`CPC .......... G06F 1111451 (2013.01); G06F 3/065
`CPC
`
` G06F 11/1451 (2013.01); G06F 3/065
`(2013.01); G06F 3/067 (2013.01);
`(2013.01); G06F 3/067 (2013.01);
`(Continued)
`(Continued)
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`A computer system maintains
`identifiers that
`identify
`A computer system maintains identifiers that identify
`changed blocks of virtual machine (VM) storage. The com(cid:173)
`changed blocks of virtual machine (VM) storage. The com-
`puter system accesses a stable VM checkpoint comprising a
`puter system accesses a stable VM checkpoint comprising a
`restorable VM image at a time, and that stores a represen(cid:173)
`restorable VM image at a time, and that stores a represen-
`tation of data of at least one block as it existed at the time.
`tation of data of at least one block as it existed at the time.
`The computer system converts the checkpoint to a reference
`The computer system converts the checkpoint to a reference
`point. Reference point information is transferable with the
`point. Reference point information is transferable with the
`VM, such that if the VM is moved to a different computing
`VM, such that if the VM is moved to a different computing
`system, any data identified by the reference point is recov(cid:173)
`system, any data identified by the reference point is recov-
`erable. The conversion includes querying the storage to
`erable. The conversion includes querying the storage to
`determine an identifier corresponding to the block of the
`determine an identifier corresponding to the block of the
`checkpoint at the time, storing this identifier as a part of the
`checkpoint at the time, storing this identifier as a part of the
`reference point, and releasing the representation of the data
`reference point, and releasing the representation of the data
`(Continued)
`(Continued)
`
`Computer System 1Q1
`Computer System 111
`102
`103
`5 102
`f 103
`
`Memory
`
`88
`I Communications Module j 104
`
`Communications Module
`
`104
`
`105
`Checkpoint Generating Module H'104
`
`SCheckpoint S' ' 06
`
`Checkpoint Accessing Module b^107
`
`Query Generating Module
`
`f-" ,08
`
`VM Reference Paint
`Generating Module
`
`, log
`
`13
`
`Query
`
`f 114
`Data
`Storage
`dentifiers
`
`( VM Reference (" 110
`-------
`)
`Point
`
`
`---------------------
`
`StateManagemen!Module r111
`f""fn
`State Management Module
`
`Replication Module
`Replication Module
`
`/12
`r112
`
`C_
`
`Data Store fil
`Data Store 11,5
`116
`111e
`Checkpoint Data
`Checkpoint Data
`
`Data Storage
`Identifiers
`Identifier
`114
`
`~
`S State Data S
`118, -' 117
`SStorage DataS
`~
`S Date Storage S
`' ' ' ' ' ' ,
`
`WIZ, Inc. EXHIBIT - 1071
`WIZ, Inc. v. Orca Security LTD.
`
`WIZ, Inc. EXHIBIT - 1071
`WIZ, Inc. v. Orca Security LTD.
`
`

`

`US 9,875,160 B2
`US 9,875,160 B2
`Page 2
`Page 2
`
`of the block from the checkpoint. The computer system then
`of the block from the checkpoint. The computer system then
`uses the reference point to identify changes in the blocks of
`uses the reference point to identify changes in the blocks of
`the storage since the time.
`the storage since the time.
`
`20 Claims, 3 Drawing Sheets
`20 Claims, 3 Drawing Sheets
`
`(52) U.S. Cl.
`(52) U.S. Cl.
`CPC ........ G06F 3/0619 (2013.01); G06F 9/45558
`G06F 3/0619 (2013.01); G06F 9/45558
`CPC
`(2013.01); G06F 2201/84 (2013.01)
`(2013.01); G06F 2201/84 (2013.01)
`( 58) Field of Classification Search
`(58) Field of Classification Search
`USPC
`....................................... 717/101-103; 718/1
`USPC
`717/101-103; 718/1
`See application file for complete search history.
`See application file for complete search history.
`
`(56)
`(56)
`
`References Cited
`References Cited
`
`U.S. PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`
`6,795,966 Bl
`6,795,966
`B1
`6,981,114 Bl
`6,981,114
`B1
`7,162,662 Bl
`7,162,662
`B1
`7,249,262 B2
`7,249,262
`B2
`7,328,366 B2
`7,328,366
`B2
`7,447,854 Bl
`7,447,854
`B1
`7,516,286 Bl
`7,516,286
`B1
`7,519,858 B2
`7,519,858
`B2
`7,574,459 B2
`7,574,459
`B2
`7,860,837 B2
`7,860,837
`B2
`7,865,470 B2
`7,865,470
`B2
`8,055,613 Bl
`8,055,613
`B1
`8,145,601 B2
`8,145,601
`B2
`8,335,902 Bl
`8,335,902
`B1
`8,356,013 B2
`8,356,013
`B2
`8,356,148 B2
`8,356,148
`B2
`8,443,166 B2
`8,443,166
`B2
`8,463,747 B2
`8,463,747
`B2
`8,463,749 B2
`8,463,749
`B2
`8,468,609 B2
`8,468,609
`B2
`8,538,919 Bl
`8,538,919
`B1
`8,560,855 B2
`8,560,855
`B2
`8,712,970 Bl
`8,712,970
`B1
`8,726,127 B2
`8,726,127
`B2
`8,726,273 B2
`8,726,273
`B2
`8,751,515 Bl
`8,751,515
`B1
`8,782,086 B2
`8,782,086
`B2
`8,788,773 B2
`8,788,773
`B2
`8,805,788 B2
`8,805,788
`B2
`8,813,204 B2
`8,813,204
`B2
`8,966,341 B2
`8,966,341
`B2
`9,081,714 B2
`9,081,714
`B2
`9,081,715 B2
`9,081,715
`B2
`9,086,994 B2
`9,086,994
`B2
`9,130,957 B2
`9,130,957
`B2
`9,276,980 B2
`9,276,980
`B2
`9,430,272 B2 *
`9,430,272
`B2 *
`2008/0126445 Al
`2008/0126445
`Al
`2008/0140963 Al
`2008/0140963
`Al
`2010/0049929 Al
`2010/0049929
`Al
`2010/0228913 Al
`2010/0228913
`Al
`2010/0262585 Al
`2010/0262585
`Al
`2010/0274980 Al
`2010/0274980
`Al
`2011/0167196 Al
`2011/0167196
`Al
`2012/0072659 Al
`2012/0072659
`Al
`2013/0067179 Al
`2013/0067179
`Al
`2013/0254479 Al
`2013/0254479
`Al
`2013/0290782 Al
`2013/0290782
`Al
`
`9/2004 Lim et al.
`9/2004 Lim et al.
`12/2005 Wu et al.
`12/2005 Wu et al.
`1/2007 Svarcas et al.
`1/2007 Svarcas et al.
`7/2007 Hauck et al.
`7/2007 Hauck et al.
`2/2008 Michelman
`2/2008 Michelman
`11/2008 Cannon
`11/2008 Cannon
`4/2009 Dalai et al.
`4/2009 Dalai et al.
`4/2009 Korlepara
`4/2009 Korlepara
`8/2009 Sen et al.
`8/2009 Sen et al.
`12/2010 Ali et al.
`12/2010 Ali et al.
`1/2011 Fries et al.
`1/2011 Fries et al.
`11/2011 Mu et al.
`11/2011 Mu et al.
`3/2012 Zizys et al.
`3/2012 Zizys et al.
`12/2012 Feathergill
`12/2012 Feathergill
`1/2013 Fachan et al.
`1/2013 Fachan et al.
`1/2013 Popovski et al.
`1/2013 Popovski et al.
`5/2013 Czezatke et al.
`5/2013 Czezatke et al.
`6/2013 Wahlert et al.
`6/2013 Wahlert et al.
`6/2013 Zizys et al.
`6/2013 Zizys et al.
`6/2013 Leggette
`6/2013 Leggette
`9/2013 Nielsen et al.
`9/2013 Nielsen et al.
`10/2013 Resch
`10/2013 Resch
`4/2014 Sim-Tang
`4/2014 Sim-Tang
`5/2014 Grube et al.
`5/2014 Grube et al.
`5/2014 Le
`5/2014 Le
`6/2014 Xing et al.
`6/2014 Xing et al.
`7/2014 Resch
`7/2014 Resch
`7/2014 Goodman et al.
`7/2014 Goodman et al.
`8/2014 Gross, IV et al.
`8/2014 Gross, IV et al.
`8/2014 Leggette
`8/2014 Leggette
`2/2015 Grube et al.
`2/2015 Grube et al.
`7/2015 Grube et al.
`7/2015 Grube et al.
`7/2015 Grube et al.
`7/2015 Grube et al.
`7/2015 Resch
`7/2015 Resch
`9/2015 Y amaura et al.
`9/2015 Yamaura et al.
`3/2016 Chan
`3/2016 Chan
`8/2016 Bezbaruah
`.......... G06F 9/45558
`8/2016 Bezbaruah
`G06F 9/45558
`5/2008 Michelman
`5/2008 Michelman
`6/2008 Thomason et al.
`6/2008 Thomason et al.
`2/2010 Nagarkar et al.
`2/2010 Nagarkar et al.
`9/2010 Czezatke et al.
`9/2010 Czezatke et al.
`10/2010 Rosikiewicz et al.
`10/2010 Rosikiewicz et al.
`10/2010 Stringham et al.
`10/2010 Stringham et al.
`7/2011 Scales et al.
`7/2011 Scales et al.
`3/2012 Wade et al.
`3/2012 Wade et al.
`3/2013 Paleologu et al.
`3/2013 Paleologu et al.
`9/2013 Czezatke et al.
`9/2013 Czezatke et al.
`10/2013 Chen et al.
`10/2013 Chen et al.
`
`2014/0025913 Al
`2014/0025913 Al
`2014/0040572 Al
`2014/0040572 Al
`2014/0164722 Al
`2014/0164722 Al
`2014/0236892 Al
`2014/0236892 Al
`2014/0250347 Al
`2014/0250347 Al
`2014/0337666 Al
`2014/0337666 Al
`2014/0337932 Al
`2014/0337932 Al
`2016/0179568 Al
`2016/0179568 Al
`2016/0203052 Al
`2016/0203052 Al
`
`1/2014 Fuente et al.
`1/2014 Fuente et al.
`2/2014 Kotagiri et al.
`2/2014 Kotagiri et al.
`6/2014 Garthwaite et al.
`6/2014 Garthwaite et al.
`8/2014 Blyler
`8/2014 Blyler
`9/2014 Grube et al.
`9/2014 Grube et al.
`11/2014 Resch et al.
`11/2014 Resch et al.
`11/2014 Leggette et al.
`11/2014 Leggette et al.
`6/2016 Bezbaruah
`6/2016 Bezbaruah
`7/2016 Starksetal.
`7/2016 Starks et al.
`
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`
`"Notice of Allowance Issued in U.S. Appl. No. 14/573,976", dated
`"Notice of Allowance Issued in U.S. Appl. No. 14/573,976", dated
`Jun. 7, 2016, 7 Pages.
`Jun. 7, 2016, 7 Pages.
`"Notice of Allowance Issued in U.S. Appl. No. 14/595,047", dated
`"Notice of Allowance Issued in U.S. Appl. No. 14/595,047", dated
`Mar. 14, 2016, 11 Pages.
`Mar. 14, 2016, 11 Pages.
`"Notice of Allowance Issued in U.S. Appl. No. 14/595,047", dated
`"Notice of Allowance Issued in U.S. Appl. No. 14/595,047", dated
`May 16, 2016, 9 Pages.
`May 16, 2016, 9 Pages.
`"International Search Report and Written Opinion Issued in PCT
`"International Search Report and Written Opinion Issued in PCT
`Application No. PCT/US2015/063565", dated Jun. 30, 2016, 20
`Application No. PCT/US2015/063565", dated Jun. 30, 2016, 20
`Pages.
`Pages.
`"International Search Report and Written Opinion Issued in PCT
`"International Search Report and Written Opinion Issued in PCT
`Application No. PCT/US2015/067761", dated May 24, 2016, 12
`Application No. PCT/US2015/067761", dated May 24, 2016, 12
`Pages.
`Pages.
`Ramdas, Aashish, "Resynchronization of Virtual Machines
`in
`Ramdas, Aashish, "Resynchronization of Virtual Machines in
`Hyper-V Replica", Retrieved
`from: <<https://blogs.technet.
`Hyper-V Replica", Retrieved
`from: <<https://blogs.technet.
`microsoft.com/virtualization/2013/05/ 10/resynchronization-of-vir(cid:173)
`micro soft. com/virtualization/2013/05/10/resynchronization-of-vir-
`tual-machines-in-hyper-v-replica/> >, May 10, 2013, 5 Pages.
`tual-machines-in-hyper-v-replica/>>, May 10, 2013, 5 Pages.
`Garg, et al., "Checkpoint-Restart
`for a Network of Virtual
`Garg, et al., "Checkpoint-Restart for a Network of Virtual
`Machines", In Proceedings of International Conference on Cluster
`Machines", In Proceedings of International Conference on Cluster
`Computing, IEEE, Sep. 23, 2013, 8 Pages.
`Computing, IEEE, Sep. 23, 2013, 8 Pages.
`Park, et al., "Fast and Space-Efficient Virtual Machine Checkpoint(cid:173)
`Park, et al., "Fast and Space-Efficient Virtual Machine Checkpoint-
`ing", In Proceedings of the 7th ACM SIGPLAN/SIGOPS Interna(cid:173)
`ing", In Proceedings of the 7th ACM SIGPLAN/SIGOPS Interna-
`tional Conference on Virtual Execution Environments, Mar. 9, 2011,
`tional Conference on Virtual Execution Environments, Mar. 9, 2011,
`pp. 75-85.
`pp. 75-85.
`Sinofsky, Steven, "Building the next generation file system for
`Sinofsky, Steven, "Building the next generation file system for
`Windows: ReFS", Retrieved from: <<https://blogs.msdn.microsoft.
`Windows: ReFS", Retrieved from: <<https://blogs.msdn.microsoft.
`com/b8/2012/0 l/ 16/building-the-next-generation-file-system-for(cid:173)
`com/b8/2012/01/16/building-the-next-generation-file-system-for-
`windows-refs/> >, Jan. 17, 2012, 45 Pages.
`windows-refs/>>, Jan. 17, 2012, 45 Pages.
`Werneburg, Ken, "VMware vSphere® Replication 5.5 Overview:
`Werneburg, Ken, "VMware vSphere® Replication 5.5 Overview:
`Simple and Effective Virtual Machine Protection", In Technical
`Simple and Effective Virtual Machine Protection", In Technical
`White Paper, Nov. 16, 2013, 14 Pages.
`White Paper, Nov. 16, 2013, 14 Pages.
`"Advanced Restore-Virtual Server Agent for VMware", Retrieved
`"Advanced Restore Virtual Server Agent for VMware", Retrieved
`from:
`< <https://documentation.commvault.com/commvault/v 10/
`from: <<https://documentation.commvault.com/commvault/v10/
`article?p~products/vs_ vmware/restore_adv.htrn>>, Retrieved on:
`article?p=products/vs vmware/restore adv.htm>>, Retrieved on:
`Nov. 7, 2014, 32 Pages.
`Nov. 7, 2014, 32 Pages.
`"Changed Block Tracking (CBT) on Virtual Machines (1020128)",
`"Changed Block Tracking (CBT) on Virtual Machines (1020128)",
`Retrieved from: < <https://kb. vmware.com/ selfservice/microsites/
`Retrieved from: <<https://kb.vmware.com/selfservice/microsites/
`search.do?cmd~displayKC&docType~kc&externalid~ 1020128
`search.do?cmd=displayKC&docType=kc&externalId=1020128
`&sliceid~l&docTypeID~DT_KB_l_l&dialogID~l55782394
`&sliceId=l&docTypeID=DT KB 1
`l&dialogID=155782394
`&stateid~0 0 160353152>>, Retrieved on: Nov. 7, 2014, 2 Pages.
`&stateld=0 0 160353152>>, Retrieved on: Nov. 7, 2014, 2 Pages.
`"High Availability and Data Protection with EMC Isilon Scale-Out
`"High Availability and Data Protection with EMC Isilon Scale-Out
`NAS", In White Paper, Nov. 2013, 36 Pages.
`NAS", In White Paper, Nov. 2013, 36 Pages.
`"What's New in Hyper-V for Windows Server 2012", Retrieved
`"What's New in Hyper-V for Windows Server 2012", Retrieved
`from:
`<<https://technet.microsoft.com/en-us/library/
`from:
`<<https://technet.microsoft.com/en-us/library/
`hh831410(vc=ws.ll).aspx>>, Feb. 29, 2012, 6 Pages.
`hh831410(v=ws.11).aspx>>, Feb. 29, 2012, 6 Pages.
`"Non Final Office Action Issued in U.S. Appl. No. 14/573,976",
`"Non Final Office Action Issued in U.S. Appl. No. 14/573,976",
`dated Dec. 3, 2015, 10 Pages.
`dated Dec. 3, 2015, 10 Pages.
`"International Preliminary Report on Patentability Issued in PCT
`"International Preliminary Report on Patentability Issued in PCT
`Application No. PCT/US2015/063565", dated Mar. 24, 2017, 14
`Application No. PCT/US2015/063565", dated Mar. 24, 2017, 14
`Pages.
`Pages.
`Notice of Allowance dated Oct. 6, 2016 cited in U.S. Appl. No.
`Notice of Allowance dated Oct. 6, 2016 cited in U.S. Appl. No.
`14/595,047.
`14/595,047.
`"SecondWritten Opinion Issued in PCT Application No. PCT/
`"SecondWritten Opinion Issued in PCT Application No. PCT/
`US2015/063565", dated Jan. 13, 2017, 12 Pages.
`US2015/063565", dated Jan. 13, 2017, 12 Pages.
`
`* cited by examiner
`* cited by examiner
`
`

`

`e •
`
`lualud °S11
`
`00
`•
`~
`~
`~
`
`~ =
`
`~
`
`8102 `£Z .11Br
`
`~
`~
`?
`N
`'"~
`N
`
`QO
`
`0 ....
`
`rJJ
`
`£ jo 1 WIN
`
`=-('D
`('D .....
`....
`....
`
`0
`
`~
`
`Zll 091`CL8`6 Sfl
`
`_."-0
`00
`-....l
`__ u.
`"'""'
`0--,
`
`= = N
`
`d r.,;_
`
`Computer System 101
`Computer System 101
`T
`102
`T
`103
`102
`103
`
`Processor B
`
`Processor
`
`Memory
`
`Communications Module
`Communications Module
`
`b i 104
`1104
`
`Checkpoint Generating Module 1 105 I
`Checkpoint Generating Module
`f-'105
`
`
`I
`
`01
`
`00--\L
`100'-J..
`~~
`
`Data Store 115
`Data Store 115
`y 116
`5116
`Checkpoint Data
`Checkpoint Data
`I \ ~ State Data )
`\ S State Data
`118-L x ' 117
`118
`""l. 117
`I
`I
`) Storage Data
`Storage Data
`
`I
`
`\
`
`Data Storage
`Identifiers
`Identifiers
`___J
`1 114
`114
`
`4
`I
`I
`I
`I
`I
`I
`I
`/
`I
`/
`
`/
`,, /,, /
`.....
`..-
`
`Checkpoint
`
`106
`106
`
`I
`
`Checkpoint Accessing Module
`
`-,‘ 107
`
`113
`
`Query
`
`+
`Checkpoint Accessing Module I- 107 I ~ I ) Data Storage
`
`Query Generating Module
`Query Generating Module
`
`108
`r108
`
`VM Reference Point
`Generating Module
`Generating Module
`
`1
`
`J 114
`114
`Data
`Data
`Storage \
`..* ______---1 Identifiers
`Identifiers
`
`VM Reference Point r109 L--1 Storage
`
`4
`110
`VM ~~~ence ~ ~ 1~ _____ ~ _________________
`(VM Reference
`Point
`
`State Management Module 1 111 I
`
`State Management Module
`
`Figure 1
`Figure 1
`
`Replication Module
`Replication Module
`
`112
`r112
`
`I
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan.23,2018
`Jan. 23, 2018
`
`Sheet 2 of 3
`Sheet 2 of 3
`
`US 9,875,160 B2
`US 9,875,160 B2
`
`2(,0 z
`
`210 -m
`
`Access Stable Virtual Machine Checkpoint That Includes
`Access Stable Virtual Machine Checkpoint That Includes
`Underlying Data Stored In Data Storage
`Underlying Data Stored In Data Storage
`
`220 --)
`220,
`
`1 '
`
`Query Data Storage To Determine Data Storage Identifiers That
`Query Data Storage To Determine Data Storage Identifiers That
`Reference The Point In Time Associated With The Checkpoint
`Reference The Point In Time Associated With The Checkpoint
`
`230,
`230— )
`
`7
`
`1 '
`
`Store Determined Data Storage Identifiers
`Store Determined Data Storage Identifiers
`As Virtual Machine Reference Point
`As Virtual Machine Reference Point
`
`Figure 2
`Figure 2
`
`3(0 _z
`
`310,
`310 --)
`
`Establish Stable, Unchanging State Within Virtual Machine
`Establish Stable, Unchanging State Within Virtual Machine
`
`320 —..
`320,
`
`1 '
`
`Access Previously Generated Reference Points To Identify
`Access Previously Generated Reference Points To Identify
`Differences In Virtual Machine State
`Differences In Virtual Machine State
`
`330,
`330 -m
`
`1 '
`
`Replicate Differences In Virtual Machine State Between Current Stable
`Replicate Differences In Virtual Machine State Between Current Stable
`State And Selected Past Stable Point In Time
`State And Selected Past Stable Point In Time
`
`Figure 3
`Figure 3
`
`

`

`e •
`
`lualud °S11
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`8102 `£Z .11Br
`
`~
`~
`?
`N
`'"~
`N
`
`QO
`
`0 ....
`
`£ jo £ WIN
`
`rJJ =-('D
`('D .....
`0 ....
`
`~
`
`~
`
`Zll 091`CL8`6 Sfl
`
`_."-0
`00
`-....l __ u.
`"'""'
`
`0--, = = N
`
`d r.,;_
`
`Data Store 408
`Data Store 408
`
`400
`~ 4~~
`
`Raw Data
`
`~ Raw Data
`
`Initial Backup
`Initial Backup
`
`Full Backup
`Full Backup
`
`Differential
`Differential
`
`Merged Backup
`Merged Backup
`
`401
`)401
`
`I Processor I B
`r404 <==>
`
`I
`
`I
`
`Computer System 420
`Computer System 420
`J 405
`J- 402
`405
`402
`Processor
`Memory
`
`411
`Backup Generating Module r411
`Backup Generating Module
`
`I
`
`..----
`,...._
`
`--.
` .. _,403
`403
`Local Disk
`Local Disk
`~ Data
`~
`Data
`401-r j - 409
`401.r
`409
`Identifiers
`Identifiers S
`
`/
`412
`---^
`VM Reference 412
`VM Reference
`Point
`)
`Point
`
`H404
`
`H410
`r410
`
`406
`r406
`
`407
`r407
`
`Figure 4
`Figure 4
`
`

`

`US 9,875,160 B2
`US 9,875,160 B2
`
`1
`1
`EFFICIENTLY PROVIDING VIRTUAL
`EFFICIENTLY PROVIDING VIRTUAL
`MACHINE REFERENCE POINTS
`MACHINE REFERENCE POINTS
`
`CROSS REFERENCE TO RELATED
`CROSS REFERENCE TO RELATED
`APPLICATION
`APPLICATION
`
`This application is a continuation of U.S. patent applica(cid:173)
`This application is a continuation of U.S. patent applica-
`tion Ser. No. 14/573,976, filed Dec. 17, 2014, and entitled
`tion Ser. No. 14/573,976, filed Dec. 17, 2014, and entitled
`"EFFICIENTLY PROVIDING VIRTUAL MACHINE
`"EFFICIENTLY PROVIDING VIRTUAL MACHINE
`REFERENCE POINTS," the entire contents of which are 10
`10
`REFERENCE POINTS," the entire contents of which are
`incorporated by reference herein in their entirety.
`incorporated by reference herein in their entirety.
`
`BACKGROUND
`BACKGROUND
`
`2
`2
`computer system accesses previously generated reference
`computer system accesses previously generated reference
`points to identify differences
`in virtual machine state
`points to identify differences in virtual machine state
`between the current stable state and a selected past stable
`between the current stable state and a selected past stable
`point in time. The computer system also replicates the
`point in time. The computer system also replicates the
`5 differences in virtual machine state between the current
`5 differences in virtual machine state between the current
`stable state and a selected past stable point in time. The
`stable state and a selected past stable point in time. The
`differences may be replicated to a data storage device as an
`differences may be replicated to a data storage device as an
`incremental backup, or may be used for remote replication
`incremental backup, or may be used for remote replication
`or disaster recovery purposes.
`or disaster recovery purposes.
`This Summary is provided to introduce a selection of
`This Summary is provided to introduce a selection of
`concepts in a simplified form that are further described
`concepts in a simplified form that are further described
`below in the Detailed Description. This Summary is not
`below in the Detailed Description. This Summary is not
`intended to identify key features or essential features of the
`intended to identify key features or essential features of the
`15 claimed subject matter, nor is it intended to be used as an aid
`15 claimed subject matter, nor is it intended to be used as an aid
`in determining the scope of the claimed subject matter.
`in determining the scope of the claimed subject matter.
`Additional features and advantages will be set forth in the
`Additional features and advantages will be set forth in the
`description which follows, and in part will be apparent to
`description which follows, and in part will be apparent to
`one of ordinary skill in the art from the description, or may
`one of ordinary skill in the art from the description, or may
`20 be learned by the practice of the teachings herein. Features
`20 be learned by the practice of the teachings herein. Features
`and advantages of embodiments described herein may be
`and advantages of embodiments described herein may be
`realized and obtained by means of the instruments and
`realized and obtained by means of the instruments and
`combinations particularly pointed out in the appended
`combinations particularly pointed out in the appended
`claims. Features of the embodiments described herein will
`claims. Features of the embodiments described herein will
`25 become more fully apparent from the following description
`25 become more fully apparent from the following description
`and appended claims.
`and appended claims.
`
`Computing systems have become ubiquitous, ranging
`Computing systems have become ubiquitous, ranging
`from small embedded devices to phones and tablets to PCs
`from small embedded devices to phones and tablets to PCs
`and backend servers. Each of these computing systems is
`and backend servers. Each of these computing systems is
`designed to process software code. The software allows
`designed to process software code. The software allows
`users to perform functions, interacting with the hardware
`users to perform functions, interacting with the hardware
`provided by the computing system. In some cases, these
`provided by the computing system. In some cases, these
`computing systems allow users to establish and run virtual
`computing systems allow users to establish and run virtual
`machines. These virtual machines may provide functionality
`machines. These virtual machines may provide functionality
`not provided by the host operating system, or may comprise
`not provided by the host operating system, or may comprise
`a different operating system altogether. In this manner,
`a different operating system altogether. In this manner,
`virtual machines may be used to extend the functionality of
`virtual machines may be used to extend the functionality of
`the computing system. Virtual machines may be backed up
`the computing system. Virtual machines may be backed up
`on virtual storage devices which themselves may be backed
`on virtual storage devices which themselves may be backed
`up to physical or virtual storage devices. Virtual machine
`up to physical or virtual storage devices. Virtual machine
`hosts may also be configured to take snapshots which
`hosts may also be configured to take snapshots which
`represent point-in-time images of the virtual machine. The 30
`represent point-in-time images of the virtual machine. The
`30
`VM snapshots or "checkpoints" include CPU state, memory
`VM snapshots or "checkpoints" include CPU state, memory
`state, storage state and other information necessary to com(cid:173)
`state, storage state and other information necessary to com-
`pletely recreate or restore the virtual machine to that point in
`pletely recreate or restore the virtual machine to that point in
`time.
`time.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`BRIEF SUMMARY
`BRIEF SUMMARY
`
`To further clarify the above and other features of the
`To further clarify the above and other features of the
`embodiments described herein, a more particular description
`embodiments described herein, a more particular description
`will be rendered by reference to the appended drawings. It
`will be rendered by reference to the appended drawings. It
`is appreciated that these drawings depict only examples of
`is appreciated that these drawings depict only examples of
`the embodiments described herein and are therefore not to
`the embodiments described herein and are therefore not to
`35 be considered limiting of its scope. The embodiments will be
`35 be considered limiting of its scope. The embodiments will be
`described and explained with additional specificity and
`described and explained with additional specificity and
`detail through the use of the accompanying drawings in
`detail through the use of the accompanying drawings in
`which:
`Embodiments described herein are directed to establish-
`Embodiments described herein are directed to establish-
`which:
`ing efficient virtual machine reference points and to speci(cid:173)
`ing efficient virtual machine reference points and to speci-
`FIG. 1 illustrates a computer architecture
`in which
`FIG. 1 illustrates a computer architecture in which
`fying a virtual machine reference point to query incremental 40
`fying a virtual machine reference point to query incremental
`embodiments described herein may operate including estab(cid:173)
`40 embodiments described herein may operate including estab-
`changes. As used herein, virtual machine reference points
`changes. As used herein, virtual machine reference points
`lishing efficient virtual machine reference points.
`lishing efficient virtual machine reference points.
`allow computer systems to identify incremental changes
`allow computer systems to identify incremental changes
`FIG. 2 illustrates a flowchart of an example method for
`FIG. 2 illustrates a flowchart of an example method for
`from specific points in time on. For example, in one embodi(cid:173)
`from specific points in time on. For example, in one embodi-
`establishing efficient virtual machine reference points.
`establishing efficient virtual machine reference points.
`ment, a computer system accesses a stable virtual machine
`ment, a computer system accesses a stable virtual machine
`FIG. 3 illustrates a flowchart of an example method for
`FIG. 3 illustrates a flowchart of an example method for
`checkpoint that includes portions of underlying data stored 45
`checkpoint that includes portions of underlying data stored
`specifying a virtual machine reference point to query incre(cid:173)
`45 specifying a virtual machine reference point to query incre-
`in data storage, where the checkpoint is associated with a
`in data storage, where the checkpoint is associated with a
`mental changes.
`mental changes.
`specific point in time. The computer system then queries the
`specific point in time. The computer system then queries the
`FIG. 4 illustrates a computer architecture
`in which
`FIG. 4 illustrates a computer architecture in which
`data storage to determine data storage identifiers that refer(cid:173)
`data storage to determine data storage identifiers that refer-
`embodiments may operate including specifying a virtual
`embodiments may operate including specifying a virtual
`ence the point in time associated with the checkpoint and
`ence the point in time associated with the checkpoint and
`machine reference point to query incremental changes.
`machine reference point to query incremental changes.
`stores the determined data storage identifiers as a virtual 50
`stores the determined data storage identifiers as a virtual so
`machine reference point or virtual machine reference point
`machine reference point or virtual machine reference point
`artifacts, where each subsequent change to the data storage
`artifacts, where each subsequent change to the data storage
`Embodiments described herein are directed to establish(cid:173)
`results in an update to the data storage identifier, so that
`Embodiments described herein are directed to establish-
`results in an update to the data storage identifier, so that
`virtual machine reference point is usable to identify incre(cid:173)
`ing efficient virtual machine reference points and to speci(cid:173)
`ing efficient virtual machine reference points and to speci-
`virtual machine reference point is usable to identify incre-
`mental changes from specific points in time on. Virtual 55
`fying a virtual machine reference point to query incremental
`55 fying a virtual machine reference point to query incremental
`mental changes from specific points in time on. Virtual
`changes. In one embodiment, a computer system accesses a
`machine reference point artifacts allow for cases where a
`machine reference point artifacts allow for cases where a
`changes. In one embodiment, a computer system accesses a
`stable virtual machine checkpoint that includes portions of
`virtual machine has two ( or more) virtual disks. Each virtual
`virtual machine has two (or more) virtual disks. Each virtual
`stable virtual machine checkpoint that includes portions of
`underlying data stored in data storage, where the checkpoint
`disk may have a different identifier for the same point in
`disk may have a different identifier for the same point in
`underlying data stored in data storage, where the checkpoint
`is associated with a specific point in time. The computer
`time, and the reference point artifact allows the computer
`time, and the reference point artifact allows the computer
`is associated with a specific point in time. The computer
`system then queries the data storage to determine data
`system to associate both of those points in time as a common 60
`system to associate both of those points in time as a common
`60 system then queries the data storage to determine data
`storage identifiers that reference the point in time associated
`point. This will be explained further below.
`point. This will be explained further below.
`storag