THE NATION MULTIROTTUK
`
`US009811662B2
`
`( 12 ) United States Patent
`Sharpe et al .
`
`( 10 ) Patent No . :
`( 45 ) Date of Patent :
`
`US 9 , 811 , 662 B2
`* Nov . 7 , 2017
`
`( 54 ) PERFORMING ANTI - VIRUS CHECKS FOR A
`DISTRIBUTED FILESYSTEM
`( 71 ) Applicant : Panzura , Inc . , Campbell , CA ( US )
`( 72 ) Inventors : Richard Sharpe , Mountain View , CA
`( US ) ; Randy Yen - pang Chou , San
`Jose , CA ( US )
`( 73 ) Assignee : PANZURA , INC . , Campbell , CA ( US )
`Subject to any disclaimer , the term of this
`( * ) Notice :
`patent is extended or adjusted under 35
`U . S . C . 154 ( b ) by 384 days .
`This patent is subject to a terminal dis -
`claimer .
`( 21 ) Appl . No . : 14 / 019 , 212
`( 22 ) Filed :
`Sep . 5 , 2013
`Prior Publication Data
`( 65 )
`US 2014 / 0007239 A1 Jan . 2 , 2014
`
`Related U . S . Application Data
`( 63 ) Continuation - in - part of application No . 13 / 971 , 621 ,
`filed
`on Aug .
`20 ,
`2013 ,
`which
`is
`a
`continuation - in - part of application No . 12 / 772 , 806 ,
`filed on May 3 , 2010 , now Pat . No . 8 , 719 , 444 , and a
`continuation - in - part of application No . 13 / 782 , 729 ,
`filed on Mar . 1 , 2013 , which is a continuation - in - part
`of application No . 13 / 769 , 185 , filed on Feb . 15 , 2013 ,
`which is a continuation - in - part of application No .
`13 / 725 , 738 , filed on Dec . 21 , 2012 , now Pat . No .
`8 , 799 , 413 , which is
`a
`continuation - in - part of
`application No . 12 / 772 , 927 , filed on May 3 , 2010 ,
`now Pat . No . 8 , 341 , 363 , and a continuation - in - part of
`application No . 13 / 225 , 194 , filed on Sep . 2 , 2011 ,
`( Continued )
`
`( 52 )
`
`( 58 )
`
`( 56 )
`
`( 51 ) Int . Cl .
`( 2006 . 01 )
`G06F 15 / 16
`( 2013 . 01 )
`G06F 21 / 56
`GO6F 1730
`( 2006 . 01 )
`U . S . CI .
`CPC . . . . . G06F 21 / 561 ( 2013 . 01 ) ; G06F 17 / 30097
`( 2013 . 01 ) ; G06F 17 / 30159 ( 2013 . 01 ) ; G06F
`17730203 ( 2013 . 01 )
`Field of Classification Search
`CPC . . . . . . . . . . . . . H04L 67 / 2842 ; H04L 67 / 1097 ; H04L
`67 / 2847 ; HO4L 29 / 06918 ; HO4L 63 / 145 ;
`GO6F 17 / 30902 ; G06F 17 / 30203
`USPC . . . . . . . . . . . .
`. . . . . . . . 709 / 217
`See application file for complete search history .
`References Cited
`U . S . PATENT DOCUMENTS
`6 , 466 , 978 B1 10 / 2002 Mukherjee et al .
`6 , 697 , 846 B1
`2 / 2004 Soltis
`( Continued )
`Primary Examiner — SM Rahman
`( 74 ) Attorney , Agent , or Firm — Park , Vaughan , Fleming
`& Dowler LLP ; Mark Spiller
`( 57 )
`ABSTRACT
`The disclosed embodiments disclose techniques that facili
`tate the process of performing anti - virus checks for a dis
`tributed filesystem . Two or more cloud controllers collec
`tively manage distributed filesystem data that is stored in
`one or more cloud storage systems ; the cloud controllers
`ensure data consistency for the stored data , and each cloud
`controller caches portions of the distributed filesystem .
`During operation , a cloud controller receives a write request
`from a client system that seeks to store a target file in the
`distributed system . A scan is then performed for this target
`file . For instance , the scan may be an anti - virus scan that
`ensures that viruses are not spread to the distributed file
`system or the clients of the distributed filesystem .
`20 Claims , 60 Drawing Sheets
`
`SITE # 1
`
`CLIENT
`
`CLIENT
`4200
`
`CONTROLLER
`
`CLOUD
`CONTROLLER
`4202
`
`CLOUD STORAGE PROVIDER ENVIRONMENT
`4210
`
`CLOUD COMPUTE SERVICES 4212
`
`CLOUD
`STORAGE
`SYSTEM
`302
`
`CLOUD
`CONTROLLER
`( VM )
`4208
`
`ANTI
`VIRUS
`( VM )
`4214
`
`CLOUD
`CONTROLLER
`4204
`
`CLOUD
`CONTROLLER
`4296
`
`Page 1 of 116
`
`Netskope Exhibit 1006
`
`

`

`US 9 , 811 , 662 B2
`Page 2
`
`( 56 )
`
`Related U . S . Application Data
`now Pat . No . 8 , 356 , 016 , and a continuation - in - part of
`application No . 13 / 295 , 844 , filed on Nov . 14 , 2011 ,
`now Pat . No . 8 , 788 , 628 .
`References Cited
`U . S . PATENT DOCUMENTS
`7 , 069 , 307 B1
`6 / 2006 Lee et al .
`7 , 610 , 285 B1
`10 / 2009 Zoellner et al .
`7 , 624 , 134 B2 11 / 2009 Stakutis et al .
`7 , 657 , 550 B2
`2 / 2010 Prahlad et al .
`7 , 844 , 582 B1 11 / 2010 Arbilla et al .
`7 , 865 , 873 B1
`/ 2011 Zoellner et al .
`1
`2004 / 0028068 A1 *
`2 / 2004 Kizhepat . . . . . . . . . . . . . . . G06F 3 / 0626
`370 / 420
`2004 / 0243644 Al 12 / 2004 Steere et al .
`6 / 2006 Armstrong . . . . . . . . . . . G06F 21 / 53
`2006 / 0136720 A1 *
`713 / 164
`2008 / 0140724 A1 *
`6 / 2008 Flynn . .
`G06F 1 / 183
`2010 / 0100698 A1 4 / 2010 Yang et al .
`2010 / 0114832 A1
`5 / 2010 Lillbridge et al .
`2010 / 0306488 A1 *
`12 / 2010 Stroberger . . . . . . . . . . G06F 11 / 2064
`711 / 162
`2010 / 0325377 Al 12 / 2010 Lango et al .
`2010 / 0333116 AL 12 / 2010 Prahlad et al .
`2011 / 0029730 A1 *
`2 / 2011 Durocher . . . . . . . . . . . . GO6F 12 / 0866
`711 / 114
`2011 / 0276713 AL 11 / 2011 Brand
`2012 / 0011176 AL
`1 / 2012 Aizman
`2012 / 0089781 A1 4 / 2012 Ranade et al .
`2013 / 0204849 AL
`8 / 2013 Chacko
`* cited by examiner
`
`Page 2 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 1 of 60
`
`US 9 , 811 , 662 B2
`
`CLIENT
`100
`
`CLIENT
`102
`
`CLIENTI
`
`A
`
`NAS DEVICE
`110
`
`NAS DEVICE
`112
`
`NAS DEVICE
`114
`
`NAS
`: Y NAS DEVICE
`NAS DEVICE
`FIG . 1A
`
`CLIENT
`100
`
`CLIENT
`102
`
`NAS DEVICE
`110
`
`NAS DEVICE
`112
`
`LOAD
`BALANCER
`120
`
`NAS DEVICE
`114
`
`FIG . 1B
`
`CLIENT En BLOCK
`
`COMPUTING
`DEVICE
`200
`
`SANI
`ISCSI
`LINK
`
`FILESYSTEM
`202
`FIG . 2
`
`STORAGE
`DEVICE
`204
`
`Page 3 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 2 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD FILES 318
`
`
`BLOCK IN CLOUD FILE 320
`DATA
`
`CLOUD STORAGE SYSTEM 302
`
`FIG . 3
`
`CLOUD CONTROLLER 300
`
`REQUEST SERVER 304
`
`CLIENT 306
`
`
`
`
`
`TRANSACTIONAL FILESYSTEM 308
`
`ROOT
`
`FILESYSTEM METADATA 310
`STORAGE 312
`
`CVA & OFFSET
`
`FILEX [ B ] [ B ] [ BX ] BLOCK PTR
`BLOCK RECORD 314
`
`DISK BLOCKS 316
`
`Page 4 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 3 of 60
`
`US 9 , 811 , 662 B2
`
`-
`
`-
`
`-
`
`CLOUD FILES 318
`
`CLOUD STORAGE SYSTEM 302
`
`FIG . 4A FIG . 4A
`
`CLOUD CONTROLLER 300
`Here
`
`REQUEST SERVER 304
`
`CLIENT 306
`
`MODIFIED METADATA 400
`
`B1 182 . . . B BLOCK PTR
`
`
`FILEX
`
`CVA & OFFSET
`
`BLOCK RECORD 314
`
`ROOT
`
`
`
`TRANSACTIONAL FILESYSTEM 308
`
`FILESYSTEM METADATA 310
`STORAGE 312
`
`DISK BLOCKS 316
`
`Page 5 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 4 of 60
`
`US 9 , 811 , 662 B2
`
`ROOT
`
`MODIFIED METADATA 400
`
`BE2 . . . BX
`
`DOW !
`
`FILEX
`
`CLOUD STORAGE SYSTEM 302
`
`VIRTUAL CLOUD FILE 412
`
`BLOCK ADDR
`
`
`BLOCK ADDR
`
`BLOCK ADDR BLOCK ADDR
`
`DISK BLOCKS 316
`
`FIG . 4B
`
`FILESYSTEM METADATA 310
`
`STORAGE 312
`
`GET OVERLAY METADATA 410
`
`
`
`
`
`TRANSACTIONAL FILESYSTEM 308
`
`Page 6 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 5 of 60
`
`US 9 , 811 , 662 B2
`
`. .
`
`40 .
`
`B12 . . . ex
`
`
`
`UPDATE META - DATA
`
`FLEX
`
`ROOT
`
`
`
`TRANSACTIONAL FILESYSTEM
`
`FILESYSTEM METADATA
`STORAGE
`
`CLOUD CONTROLLER 420
`
`CLOUD CONTROLLER
`
`300
`
`.
`
`SNAPSHOT
`
`TRANSACTIONAL FILESYSTEM 308
`
`FILET SNAPSHOTS
`5
`CLOUD
`
`NOTIFICATION
`FILESYSTEM METADATA 310
`STORAGE 312
`
`
`
`
`
`FIG . 4C
`
`CLOUD STORAGE SYSTEM 302
`
`Page 7 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 6 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD FILES 318
`
`CLOUD STORAGE SYSTEM 302
`
`FIG . 4D
`
`CLOUD CONTROLLER 420
`
`FILESYSTEM METADATA 424
`
`TRANSACTIONAL FILESYSTEM
`REQUEST
`a rtet SERVER
`STORAGE 426
`
`ROOT
`
`. ' FLEX
`
`CVA & OFFSET
`
`[ B ] [ B ] [ B ] BLOCKPERS
`BLOCK RECORD
`
`CLIENT 421 I am the st
`
`DISK BLOCKS 428
`
`Page 8 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 7 of 60
`
`US 9 , 811 , 662 B2
`
`DATA BLOCKS 432
`HHH
`
`. .
`
`olton
`LOOKUP
`
`L2
`
`L1 HASH TABLE
`
`ROOT
`
`H
`
`OU
`LOOKUP OPERATION
`
`FILEX
`
`Ex ] - - - 62 ] D1
`
`BLOCK PTR
`
`CVA & OFFSETY
`
`BLOCK RECORD 436
`
`FIG . 4E Gen
`
`CLOUD STORAGE SYSTEM 302
`
`PRC 430
`
`LOOKUP STRUCTURE 434
`FILESYSTEM METADATA 424
`
`
`
`TRANSACTIONAL FILESYSTEM
`
`CLOUD CONTROLLER 420
`
`Page 9 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 8 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`RECEIVE IN A CLOUD CONTROLLER A REQUEST TO
`ACCESS A DATA BLOCK FOR A FILE
`500
`
`TRAVERSE THE METADATA TO IDENTIFY A METADATA
`ENTRY THAT IS ASSOCIATED WITH THE DATA BLOCK
`510
`
`USE METADATA ENTRY TO DOWNLOAD A CLOUD FILE
`CONTAINING THE DATA BLOCK FROM A CLOUD
`STORAGE SYSTEM
`520
`
`DETERMINE THAT AN ADDITONAL CLOUD FILE IN THE
`CLOUD STORAGE SYSTEM INCLUDES DATA THAT IS
`LIKELY TO BE ACCESSED IN CONJUNCTION WITH THE
`DATA BLOCK
`530
`
`PRE - FETCH THE ADDITIONAL CLOUD FILE FROM THE
`CLOUD STORAGE SYSTEM
`540
`
`CEND END
`
`FIG . 5
`
`Page 10 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 9 of 60
`
`US 9 , 811 , 662 B2
`
`
`
`CLOUD STORAGE SYSTEM 302
`
`FIG . 6A
`
`COMPUTING DEVICE 600
`
`.
`
`.
`
`.
`
`. .
`
`FILESYSTEM DEVICE
`DRIVER 616
`
`REQUEST SERVER 608
`
`
`
`
`
`NAS FILESYSTEM 642
`
`STORAGE 644
`
`NAS
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`. .
`
`.
`
`.
`
`.
`
`.
`
`-
`
`. *
`
`-
`
`.
`
`-
`
`.
`
`-
`
`.
`
`-
`
`.
`
`. -
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`DEVICE 640 mage
`COMPUTING DEVICE 630
`
`. .
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`»
`
`I
`
`
`
`STORAGE MANAGEMENT SYSTEM 632
`
`CLIENT 610
`
`CLIENT 612
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`
`
`
`
`LOCAL COMPUTING ENVIRONMENT 614
`
`. .
`
`.
`
`.
`
`.
`
`. .
`
`.
`
`-
`
`-
`
`- -
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- -
`
`-
`
`-
`
`- -
`
`-
`
`-
`
`Page 11 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 10 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD STORAGE SYSTEM 302
`
`FIG . 6B
`
`COMPUTING DEVICE 600
`
`FILESYSTEM DEVICE
`DRIVER 616
`
`REQUEST SERVER 608
`
`
`
`STORAGE MANAGEMENT SYSTEM 632
`
`
`
`
`
`NAS FILESYSTEM 642
`
`STORAGE 644
`
`CLIENT 610
`
`CLIENT 612
`
`CLOUD CONTROLLER 601
`
`
`
`LOCAL COMPUTING ENVIRONMENT 614
`
`
`
`
`
`Page 12 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 11 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD STORAGE SYSTEM 302
`
`FIG . 6C
`
`FILESYSTEM DEVICE
`DRIVER 616
`
`
`
`STORAGE MANAGEMENT SYSTEM 618
`
`STORAGE 620
`
`REQUEST SERVER 608
`
`querer
`CLIENT 612
`
`CLIENT 610
`
`
`
`
`
`OPERATING SYSTEM 606 HOST OPERATING SYSTEM 604 GUEST
`
`
`
`
`
`CLOUD CONTROLLER 602
`
`
`
`
`
`LOCAL COMPUTING ENVIRONMENT 614
`
`Page 13 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 12 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`RECEIVE A REQUEST TO PERFORMAFILE
`OPERATION FROM A CLIENT COMPUTING DEVICE
`700
`
`ISSUE ONE OR MORE FILESYSTEM OPERATIONS FOR
`THE REQUEST
`710
`
`USE FILESYSTEM DEVICE DRIVER TO EXTRACT
`FILESYSTEM - LEVEL INFORMATION FROM THE ONE
`OR MORE FILESYSTEM OPERATIONS
`720
`
`FORWARD THE EXTRACTED FILESYSTEM - LEVEL
`INFORMATION TO A STORAGE MANAGEMENT
`SYSTEM
`730
`
`END
`
`FIG . 7A
`
`Page 14 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 13 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`EXECUTE A GUEST OPERATING SYSTEM ON A
`VIRTUAL MACHINE
`740
`
`RECEIVE IN THE GUEST OPERATING SYSTEMA
`REQUEST TO PERFORM A FILE OPERATION FROM A
`CLIENT COMPUTING DEVICE
`750
`
`USE FILESYSTEM DEVICE DRIVER FOR THE GUEST
`OPERATING SYSTEM TO FORWARD INFORMATION
`ASSOCIATED WITH THE REQUEST TO THE HOST
`OPERATING SYSTEM
`760
`
`USE THE FORWARDED INFORMATION IN THE HOST
`OPERATING SYSTEM TO MANAGE A FILE
`ASSOCIATED WITH THE REQUEST IN THE CLOUD
`CONTROLLER AND IN A CLOUD STORAGE SYSTEM
`770
`
`CEND END
`
`FIG . 7B
`
`Page 15 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`N ov . 7 , 2017
`
`Sheet 14 of 60
`
`US 9 , 811 , 662 B2
`
`FIG . 8
`
`CLOUD CONTROLLER 800
`
`CLOUD STORAGE SYSTEM 302
`
`MIRROR STORAGE SYSTEM 804
`
`?? CLOUD CONTROLLER 892
`
`meinbienvenieur
`
`31
`
`CLIENT
`
`CLIENT
`
`?
`
`Page 16 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 15 of 60
`
`US 9 , 811 , 662 B2
`
`COMPUTING ENVIRONMENT 900
`
`. .
`
`.
`
`.
`
`.
`
`SERVER
`930
`
`DATABASE
`970
`
`SERVER
`950
`
`VUUUUUU
`
`SERVER
`940
`
`NETWORK
`960
`
`
`
`??????? ????????
`
`.
`
`. v
`
`v .
`
`.
`
`v
`
`. .
`
`. vv
`
`. . . vv
`
`v
`
`.
`
`vv .
`
`vv . . .
`
`CLIENT
`910
`
`920
`
`CBS CB USER
`
`USER
`921
`
`M
`CLIENT
`911
`
`CLIENT
`912
`
`A
`
`W
`
`ow
`
`!
`
`APPLIANCE
`990
`
`CLOUD - BASED
`STORAGE SYSTEM
`995
`
`DEVICES
`980
`
`FIG . 9
`
`Page 17 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 16 of 60
`
`US 9 , 811 , 662 B2
`
`COMPUTING DEVICE 1000
`
`PROCESSOR
`1002
`
`STORAGE
`MECHANISM
`1004
`
`RECEIVING MECHANISM
`1006
`
`STORAGE MANAGEMENT
`MECHANISM
`1008
`
`FIG . 10
`
`Page 18 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 17 of 60
`
`US 9 , 811 , 662 B2
`
`… … … …
`
`?
`
`NEW METADATA / DATA SNAPSHOTS AND
`CLOUD FILES TO CLOUD STORAGE SYSTEM
`NEW METADATA AND DATA BEING
`DOWNLOADED TO CLOUD CONTROLLERS
`CLIENT TO CLOUD CONTROLLER
`CONNECTION
`
`1
`
`BACKUP
`CLOUD
`CONTROLLER
`100
`
`CLIENT
`1130
`
`CLOUD
`CONTROLLER
`1112
`
`?? CLIENT
`
`1114
`
`CLIENT
`1116
`
`CLOUD
`CONTROLLER
`102
`
`CLOUD
`STORAGE
`SYSTEM
`302
`
`art
`? |
`| ?? …
`
`CLOUD
`CONTROLLER
`139
`
`CLIENT
`1128
`
`CLOUD
`CONTROLLER
`1108
`
`CLIENT
`1126
`
`CLIENT
`1124
`
`CLOUD
`CONTROLLER
`1104
`
`| |
`… ??
`
`CLOUD
`CONTROLLER
`1106
`
`CLIENT
`1118
`
`CLIENT
`1122
`
`CLIENT
`1120
`
`FIG . 1A
`
`Page 19 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 18 of 60
`
`US 9 , 811 , 662 B2
`
`11 NEW METADATA / DATA SNAPSHOTS AND
`- CLOUD FILES TO CLOUD STORAGE SYSTEM
`NEW METADATA AND DATA BEING
`* DOWNLOADED TO CLOUD CONTROLLERS
`CLIENT TO CLOUD CONTROLLER
`seres sorties were 11
`CONNECTION
`
`CLIENT
`1130
`
`CLOUD
`CONTROLLER
`1112
`
`CLOUD
`CONTROLLER
`1110
`
`BACKUP
`CLOUD
`CONTROLLER
`1100
`
`In 11 -
`
`meer
`
`CLOUD
`STORAGE
`SYSTEM
`302
`
`CLIENT
`1114
`onder
`
`manier was
`
`weten
`
`CLIENT
`1116
`
`1102 X
`
`NOU
`CONTEXOLLER
`
`CLOUD
`CONTROLLER
`1104
`
`CLIENT
`1128
`
`CLIENT
`1126
`
`CLOUD
`CONTROLLER
`1108
`
`CLIENT
`
`1124 SON - -
`
`CLIENT
`1118
`
`CLIENT
`1120
`
`CLOUD
`CONTROLLER
`1106
`
`- CLIENT
`
`1122
`
`FIG . 11B
`
`Page 20 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 19 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`1200
`
`RECEIVE METADATA SNAPSHOTS DISTRIBUTED BY
`THE TWO OR MORE CLOUD CONTROLLERS ATA
`BACKUP CLOUD CONTROLLER
`1210
`
`UPON DETERMINING THE FAILURE OF A CLOUD
`CONTROLLER , RECEIVE DATA REQUESTS FROM
`CLIENTS ASSOCIATED WITH THE CLOUD
`CONTROLLER AT THE BACKUP CLOUD CONTROLLER
`1220
`
`CEND END
`
`FIG . 12
`
`Page 21 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 20 of 60
`
`US 9 , 811 , 662 B2
`
`para
`
`CLIENT
`
`CLOUD
`CONTROLLER
`1300
`
`pengeluaran
`
`CLIENT
`
`CLIENT
`
`CLOUD
`CONTROLLER
`1302
`
`CLOUD
`STORAGE
`SYSTEM
`1304
`
`CLOUD
`STORAGE
`SYSTEM
`1306
`
`FIG . 13A
`
`COB 0
`at the time to
`+ com
`
`CCID
`
`CVA FORMAT
`1310
`FILEID
`S SID
`FIG . 13B
`
`FSID
`
`OFFSET
`
`Page 22 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 21 of 60
`
`US 9 , 811 , 662 B2
`
`CLIENT
`
`CLOUD
`CONTROLLER
`
`1406 det
`
`CLOUD
`STORAGE
`SYSTEM
`1402
`
`CLIENT
`
`CLIENT
`
`CLOUD
`CONTROLLER
`
`-
`
`SCO
`
`CLOUD
`STORAGE
`SYSTEM
`1404
`
`)
`
`CLOUD F?LE
`1400
`
`FIG . 14
`
`Page 23 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 22 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`1500
`
`DISTRIBUTE DATA FOR THE DISTRIBUTED
`FILESYSTEM ACROSS TWO OR MORE CLOUD
`STORAGE SYSTEMS TO OPTIMIZE
`PERFORMANCE AND / OR COST
`1510
`
`END
`
`FIG . 15
`
`Page 24 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 23 of 60
`
`US 9 , 811 , 662 B2
`
`PAST YEARS : RECENT MONTHS : RECENT WEEKS :
`DAILY
`WEEKLY
`MONTHLY
`SNAPSHOTS
`SNAPSHOTS
`SNAPSHOTS
`
`naman
`
`RECENT DAYS :
`HOURLY
`SNAPSHOTS
`
`munt TIME
`
`SNAPSHOT
`1600
`
`SNAPSHOT
`1602
`
`DATA CREATED AND
`DELETED IN THIS
`INTERVAL
`1604
`FIG . 16A
`
`CLOUD
`CONTROLLER
`1606
`
`CLOUD
`STORAGE
`SYSTEM
`1610
`
`ARCHIVAL
`CLOUD
`STORAGE
`SYSTEM
`1612
`
`CLOUD FILE
`1608
`
`FIG . 16B
`
`Page 25 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 24 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`1700
`
`DETERMINE THAT A DATA SNAPSHOT PREVIOUSLY
`STORED IN A CLOUD STORAGE SYSTEM IS NO
`LONGER BEING ACTIVELY REFERENCED IN THE
`DISTRIBUTED FILESYSTEM
`1710
`
`TRANSFER THE DATA SNAPSHOT FROM THE CLOUD
`STORAGE SYSTEM TO AN ARCHIVAL CLOUD
`STORAGE SYSTEM
`1720
`
`END END
`
`FIG . 17
`
`Page 26 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 25 of 60
`
`US 9 , 811 , 662 B2
`
`. . . . . . . .
`
`NEW METADATA / DATA SNAPSHOTS AND
`- CLOUD FILES TO CLOUD STORAGE SYSTEM
`NEW METADATA AND DATA BEING
`DOWNLOADED TO CLOUD CONTROLLERS
`CLIENT TO CLOUD CONTROLLER
`CONNECTION
`
`CLIENT
`
`CLOUD
`CONTROLLER
`
`CLOUD
`CONTROLLER
`
`. - .
`. -
`. -
`( OPTIONAL )
`BACKUP
`CLOUD
`CONTROLLER
`1806
`
`CLOUD
`STORAGE
`SYSTEM
`302
`
`- -
`
`- -
`
`CLIENT
`1802
`
`CLIENT
`
`CLOUD
`CONTROLLER
`1800
`
`CLOUD
`CONTROLLER
`1804
`
`CLIENT
`
`CLIENT
`
`CLOUD
`CONTROLLER
`
`O .
`
`CLIENT
`
`CLIENT
`
`CLOUD
`CONTROLLER
`
`CLIENT È CLIENT
`
`FIG . 18
`
`Page 27 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 26 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`1900
`
`RECEIVE IN A CLOUD CONTROLLER A REQUEST
`FROM A CLIENT FOR A DATA BLOCK OF A FILE
`STORED IN THE DISTRIBUTED FILESYSTEM
`1910
`
`DETERMINE THAT THE REQUESTED DATA
`BLOCK IS NOT CURRENTLY CACHED
`IN THE CLOUD CONTROLLER
`1920
`
`SEND A PEER CACHE REQUEST FOR THE
`REQUESTED DATA BLOCK TO A PEER CLOUD
`CONTROLLER IN THE DISTRIBUTED FILESYSTEM
`1930
`
`C
`
`END END
`
`FIG . 19
`
`Page 28 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 27 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`A CLOUD CONTROLLER A METADATA
`MAINTAIN IN
`HIERARCHY THAT REFLECTS THE CURRENT STATE
`OF A DISTRIBUTED FILESYSTEM
`2000
`
`STORE IN THE CLOUD CONTROLLER A SET OF NEW
`DATA FOR THE DISTRIBUTED FILESYSTEM THAT WAS
`RECEIVED FROM A CLIENT
`2010
`
`CREATE A METADATA ENTRY FOR THE NEW DATA IN
`THE METADATA HIERARCHY
`2020
`
`UPDATE THE OVERLAY METADATA TO POINT TO THE
`METADATA ENTRY AND THE NEW DATA STORED IN
`THE CLOUD CONTROLLER
`2030
`
`USE THE OVERLAY METADATA TO GENERATE AN
`INCREMENTAL SNAPSHOT FOR THE NEW DATA
`2040
`
`END
`
`FIG . 20
`
`Page 29 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 28 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD
`CONTROLLER
`2100
`
`STORAGE DRIVE 2102
`
`DATA AND / OR
`EMPTY
`BLOCKS
`
`METADATA
`BLOCKS
`2104
`
`DISK
`PLATTERS
`
`FIG . 21A
`
`* * *
`
`* * * * * * *
`
`* * * * * * * * * * * * * * * * *
`
`*
`
`METADATA
`STORAGE
`REGIONS
`2112
`
`DISK
`PLATTER
`2110
`
`DATA
`STORAGE
`REGIONS
`2114
`
`FIG . 21B
`
`Page 30 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 29 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`MAINTAIN IN THE LOCAL STORAGE DEVICE OF A
`CLOUD CONTROLLER : ( 1 ) ONE OR MORE METADATA
`REGIONS CONTAINING A METADATA HIERARCHY
`THAT REFLECTS THE CURRENT STATE OF A
`DISTRIBUTED FILESYSTEM ; AND ( 2 ) CACHED DATA
`FOR THE DISTRIBUTED FILESYSTEM
`2200
`
`RECEIVE AN INCREMENTAL METADATA SNAPSHOT
`THAT REFERENCES NEW DATA WRITTEN TO THE
`DISTRIBUTED FILESYSTEM
`2210
`
`STORE UPDATED METADATA FROM THE
`INCREMENTAL METADATA SNAPSHOT IN A METADATA
`REGION ON THE LOCAL STORAGE DEVICE
`2220
`
`CEND END
`
`FIG . 22
`
`Page 31 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 30 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD STORAGE SYSTEM 302
`
`ISO
`
`0
`
`OSVO
`
`0
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`FIG . 23
`
`CLOUD FILE 2304
`
`2000 :
`
`: RELATED DATA BLOCKS 2310
`
`OOOOOO
`
`
`
`CLOUD CONTROLLER 2300
`
`CLIENT 2302
`
`DATA BLOCKS
`
`
`
`TRANSPORT TERMINATION POINT 2312
`TARGET DATA BLOCK 2306
`BLOCK METADATA 2308
`
`Page 32 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 31 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`2400
`
`RECEIVE IN A CLOUD CONTROLLERA REQUEST
`FROM A CLIENT FOR A DATA BLOCK OF A TARGET
`FILE THAT IS STORED IN THE DISTRIBUTED
`FILESYSTEM BUT NOT CURRENTLY CACHED IN THE
`CLOUD CONTROLLER
`2410
`
`INITIATE A REQUEST TO A CLOUD STORAGE SYSTEM
`FOR A CLOUD FILE CONTAINING THE REQUESTED
`DATA BLOCK
`2420
`
`USE BLOCK METADATA IN A PORTION OF THE CLOUD
`FILE THAT HAS ALREADY BEEN RECEIVED TO
`DETERMINE THE PORTIONS OF THE CLOUD FILE
`THAT SHOULD BE DOWNLOADED TO AND CACHED IN
`THE CLOUD CONTROLLER
`2430
`
`END
`
`FIG . 24
`
`Page 33 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 32 of 60
`
`US 9 , 811 , 662 B2
`
`
`
`
`
`CLOUD CONTROLLER 2506
`
`* - - -
`
`
`
`
`
`CLOUD CONTROLLER 2500
`
`CLIENT
`CLIENT
`
`CLIENT
`
`SITE LOCALITY POLICY 2516
`GLOBAL LOCALITY POLICY 2510
`
`-
`
`LOCAL LOCALITY POLICY 2512
`GLOBAL LOCALITY POLICY 2510
`
`
`
`
`
`CLOUD CONTROLLER 2508
`
`SITE LOCALITY POLICY 2516
`GLOBAL LOCALITY POLICY 2510
`
`CLIENT
`
`-
`
`- -
`
`-
`
`-
`
`CO - LOCATED CLOUD CONTROLLERS
`
`-
`
`-
`
`CLOUD STORAGE SYSTEM 302
`
`
`
`
`
`CLOUD CONTROLLER 2504
`
`PARTIAL FILESYSTEM LOCALITY POLICY 2514
`GLOBAL LOCALITY POLICY 2510
`
`.
`CLIENT
`CLIENT
`
`
`
`
`
`CLOUD CONTROLLER 2502
`
`GLOBAL LOCALITY POLICY 2510
`
`CLIENT
`
`FIG . 25
`
`Page 34 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 33 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`2600
`
`RECEIVE IN A CLOUD CONTROLLER A LOCALITY
`POLICY THAT SPECIFIES ONE OR MORE
`MANAGEMENT POLICIES FOR THE CLOUD
`CONTROLLER
`2610
`
`MANAGE DATA THAT IS CREATED , CACHED , AND / OR
`MANAGED BY THE CLOUD CONTROLLER BASED ON
`THE LOCALITY POLICY
`2620
`
`(
`
`END END
`
`FIG . 26
`
`Page 35 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 34 of 60
`
`US 9 , 811 , 662 B2
`
`CLIENT 2710
`
`FIG . 27A
`
`CLOUD STORAGE SYSTEM 302
`
`SNAPSHOT HIERARCHY 2702
`
`ACTIVE FILESYSTEM METADATA 2704
`OVERLAY METADATA 2706
`
`
`
`TRANSACTIONAL FILESYSTEM
`
`DEDUPLICATION TABLES 2716
`
`[ B ] [ B2 ] - [ BX | | •••
`FILE XI
`
`ROOT
`
`FILEY . . . O
`
`. . •
`
`2 AT
`
`AB
`
`CACHED DATA BLOCKS 2708
`
`
`
`
`
`CLOUD CONTROLLER 2700
`
`Page 36 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 35 of 60
`
`US 9 , 811 , 662 B2
`
`FIG . 27B
`
`* CLIENT 2710
`DELETE REQUEST 2712
`
`SNAPSHOT HIERARCHY 2702
`
`ACTIVE FILESYSTEM METADATA 2704
`OVERLAY METADATA 2706
`
`
`
`TRANSACTIONAL FILESYSTEM
`
`
`
`
`
`CLOUD CONTROLLER 2700
`
`CLOUD STORAGE SYSTEM 302
`
`DEDUPLICATION TABLES 2716
`
`. .
`
`. . .
`
`FILEX [ B ] [ B2 - [ BX
`
`I
`
`ROOT
`
`JELEY . O
`A1
`
`A
`
`CACHED DATA BLOCKS 2708
`
`Page 37 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 36 of 60
`
`US 9 , 811 , 662 B2
`
`CLIENT 2710
`
`FIG . 270
`
`CLOUD STORAGE SYSTEM 302
`
`DEDUPLICATION UPDATES 2714
`
`DEDUPLICATION TABLES 2716
`
`Quy LIST OF CLOUD FILES TO DELETE 2718
`
`SNAPSHOT HIERARCHY 2702
`
`ACTIVE FILESYSTEM METADATA 2704
`OVERLAY METADATA 2706
`
`
`
`TRANSACTIONAL FILESYSTEM
`
`BX ]
`
`* * . . .
`
`FILA
`
`ROOT
`
`D
`. . .
`
`FILEY . . !
`FILEY *
`) AT
`
`13
`
`CACHED DATA BLOCKS 2708
`
`
`
`
`
`CLOUD CONTROLLER 2700
`
`Page 38 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 37 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`2800
`
`RECEIVE IN A CLOUD CONTROLLER A REQUEST
`FROM A CLIENT TO DELETE A FILE IN THE
`DISTRIBUTED FILESYSTEM
`2810
`
`UPDATE A USER VIEW OF THE DISTRIBUTED
`FILESYSTEM TO PRESENT THE APPEARANCE OF THE
`TARGET FILE BEING DELETED TO THE CLIENT
`2820
`
`INITIATE A BACKGROUND DELETION OPERATION TO
`DELETE THE TARGET FILE WITHOUT NEGATIVELY
`AFFECTING THE PERFORMANCE OF THE OTHER
`USERS OF THE DISTRIBUTED FILESYSTEM
`2830
`
`END
`
`FIG . 28
`
`Page 39 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 38 of 60
`
`US 9 , 811 , 662 B2
`
`FIG . 29A
`
`DATA TO BE
`WRITTEN
`
`DATA BLOCK
`2904
`
`DATA BLOCK
`2902
`
`DATA BLOCK
`2900
`
`HASH KEYS
`2908
`
`HASH
`FUNCTION
`2906
`
`HASH TABLE
`2910
`
`?
`
`BLOCK ENTRY
`2914
`
`?
`
`?
`
`BLOCK ENTRY
`2912
`
`BLOCK ENTRY
`2916
`
`Page 40 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 39 of 60
`
`US 9 , 811 , 662 B2
`
`FIG . 29B
`
`DATA BLOCK
`2904
`DATA BLOCK
`2902
`DATA BLOCK
`2900
`
`HASH
`FUNCTION
`2906
`
`HASH KEYS
`2908
`
`HASH TABLE
`2920
`
`BLOCK
`ENTRY
`2912
`
`BLOCK
`ENTRY BLOCK
`2914
`ENTRY
`2916
`
`STRUCTURES
`2926
`
`TEMPORAL
`DEDUPLICATION
`STRUCTURE
`2922
`
`- 1 - - mo4 LOOKUP
`HINDI
`
`BLOCK
`ENTRY
`POINTERS
`2928
`
`TEMPORAL
`DEDUPLICATION
`STRUCTURE
`2924
`
`Page 41 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 40 of 60
`
`US 9 , 811 , 662 B2
`
`NEW DATA
`
`2952 = = CLIENT
`
`2954
`
`CLOUD
`CONTROLLER
`2950
`
`DEDUPLICATION
`TABLE
`2956
`
`INCREMENTAL
`DATA SNAPSHOT
`2960
`
`2958 3
`
`INCREMENTAL
`METADATA
`SNAPSHOT
`
`CLOUD
`STORAGE
`SYSTEM
`302
`
`CLOUD CONTROLLER
`2962
`
`DEDUPLICATION
`TABLE
`2964
`
`FIG . 29C
`
`Page 42 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 41 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`3000
`
`RECEIVE IN A CLOUD CONTROLLER AN
`INCREMENTAL METADATA SNAPSHOT THAT
`REFERENCES NEW DATA THAT WAS ADDED TO THE
`DISTRIBUTED FILESYSTEM BYA
`REMOTE CLOUD CONTROLLER
`3010
`
`EXTRACT A SET OF DEDUPLICATION INFORMATION
`FROM THE INCREMENTAL METADATA SNAPSHOT
`3020
`
`RECEIVE A CLIENT WRITE REQUEST AT THE CLOUD
`CONTROLLER
`3030
`
`USE THE EXTRACTED DEDUPLICATION INFORMATION
`TO DETERMINE THAT ONE OR MORE DATA BLOCKS IN
`THE CLIENT WRITE REQUEST HAVE ALREADY BEEN
`WRITTEN TO THE DISTRIBUTED FILESYSTEM
`3040
`
`(
`
`END END
`
`FIG . 30
`
`Page 43 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 42 of 60
`
`US 9 , 811 , 662 B2
`
`CLIENT
`3114
`
`VM 3120
`
`APPLICATION
`( AND / OR SERVICE )
`3122
`
`EXPORTED
`MOUNT POINT
`FOR DFS
`3110
`
`SERVER
`3100
`
`LOCAL
`STORAGE
`3106
`
`CLOUD
`CONTROLLER
`INSTANCE
`3102
`
`CACHED
`DATA
`BLOCKS
`FOR DFS
`3112
`
`VM 3104
`
`STORAGE POOL 3108
`
`CLOUD
`STORAGE
`SYSTEM
`302
`
`FIG . 31
`
`Page 44 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 43 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USEOLLEGISTRIE
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`3200
`
`EXECUTE PROGRAM INSTRUCTIONS FOR AN
`APPLICATION IN A VIRTUAL MACHINE
`3210
`
`HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
`
`STORE DATA ASSOCIATED WITH THE APPLICATION
`AND THE VIRTUAL MACHINE IN THE DISTRIBUTED
`FILESYSTEM
`3220
`
`UPON DETECTING A FAILURE , USE THE DATA
`STORED IN THE DISTRIBUTED FILESYSTEM FOR THE
`APPLICATION AND THE VIRTUAL MACHINE TO
`RECOVER AND RESUME THE EXECUTION OF THE
`VIRTUAL MACHINE AND THE APPLICATION
`3230
`
`END END
`
`FIG . 32
`
`Page 45 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 44 of 60
`
`US 9 , 811 , 662 B2
`
`STORAGE
`SYSTEM
`CLOUD
`
`302
`
`
`
`CLOUD FILE
`
`CONTAINING
`
`ADDT ' L BLOCKS
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`FIG . 33A
`
`CONTROLLER
`CLOUD
`
`3302
`
`
`
`CACHED BLOCK # 1
`
`CACHED BLOCK # 2
`
`NOT CACHED ,
`
`REQ . CLOUD FILE
`
`
`PROCESS CLOUD
`CACHED BLOCK # 3
`
`BLOCK # 3
`FILE TO ACCESS
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`
`
`REMOTE SITE
`
`CLIENT
`
`3300
`
`
`
`REQUEST BLOCK # 1
`
`REQUEST BLOCK # 3
`
`REQUEST BLOCK # 2
`
`
`
`
`REQUESTED BLOCK
`
`
`CLIENT DELAY FOR
`
`3304
`
`IME
`
`Page 46 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 45 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD
`
`FILE CONTAINING ADDT ' L BLOCKS
`
`
`
`BLOCK # 3 RECEIVED AND CACHED
`
`FIG . 33B
`
`CLOUD STORAGE SYSTEM 302
`
`CLOUD CONTROLLER 3302
`
`
`
`REMOTE SITE
`
`CLIENT 3300
`
`-
`
`*
`
`-
`
`W
`
`* W
`
`-
`
`*
`
`-
`
`*
`
`-
`
`*
`
`-
`
`W
`
`* W
`
`-
`
`*
`
`-
`
`*
`
`-
`
`*
`
`-
`
`W
`
`* W
`
`-
`
`*
`
`-
`
`*
`
`-
`
`*
`
`-
`
`WWW
`
`
`
`
`
`BLOCK # 1 CACHED ; DELAY RESPONSE + REQ . CLOUD FILES WITH
`
`AWAITING CLOUD FILE ( S ) CACHED
`
`BLOCK # 1 BLOCK # 2 CACHED ; DELAY RESPONSE WHILE
`CACHED
`ADDT ' L BLOCKS
`
`
`BLOCK # 3
`
`BLOCK # 2 BLOCK # 3 CACHED ; ADJUST RESPONSE DELAY ( OPTIONAL ) CACHED
`
`
`
`REQUEST BLOCK # 1 -
`
`
`
`DELAY INTERVAL 3304
`
`REQUEST
`
`
`
`BLOCK # 2 - DELAY INTERVAL 3306
`
`
`
`REQUEST BLOCK # 4 -
`
`REQUEST
`
`BLOCK # 3 - ADJUSTED DELAY
`INTERVAL 3308
`
`TIME
`
`Page 47 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 46 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`3400
`
`RECEIVE IN A CLOUD CONTROLLER A REQUEST
`FROM A CLIENT SYSTEM FOR A DATA BLOCK IN A
`TARGET FILE THAT IS STORED IN THE DISTRIBUTED
`FILESYSTEM
`3410
`
`DELAY THE TRANSMISSION OF THE CACHED DATA
`BLOCK FROM THE CLOUD CONTROLLER TO THE
`CLIENT SYSTEM FOR A DELAY INTERVAL
`3420
`
`CEND END
`
`FIG . 34
`
`Page 48 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 47 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD STORAGE SYSTEM 302
`
`CLOUD
`
`FILE ( S ) CONTAINING NEEDED
`DATA BLOCKS
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`CLOUD CONTROLLER 3502
`
`DETERMINE CACHE STATUS FOR
`
`
`
`DATA BLOCKS OF FILEX , REQUEST UNCACHED BLOCKS
`
`
`WRITES , UPDATE DFS METADATA : SNAPSHOT
`DEDUPLICATION OPERATIONS
`DETECT THE DUPLICATE
`METADATA TO REFLECT
`FILE Y
`
`
`
`
`TRANSFER CACHED DATA
`
`BLOCKS FOR FILE X
`
`FIG . 35A
`
`
`
`REMOTE SITE
`
`CLIENT 3500
`
`
`
`RECEIVE USER REQUEST TO COPY FILEX TO FILE Y
`
`
`
`
`REQUEST ALL DATA BLOCKS FOR FILEX
`
`
`
`
`
`WRITE DATA BLOCKS FOR
`FILE X TO FILEY
`
`TIME
`
`Page 49 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 48 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD STORAGE SYSTEM 302
`
`99 . 64999
`
`499 .
`
`0 .
`
`09 . 2009 +
`
`499
`
`
`
`REMOTE SITE + 0000 29 . . 099 .
`
`CLOUD CONTROLLER 3502
`
`CLIENT 3500
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`CLOUD
`( OPTIONAL )
`
`FILE ( S ) CONTAINING PRE - FETCHED
`DATA BLOCKS
`
`FIG . 35B
`
`( OPTIONAL ) PRE - FETCH AND CACHE ONE OR MORE BLOCKS OF FILEX
`
`METADATASNAPSHOT
`
`FILE Y
`
`
`
`GENERATE METADATA FOR
`
`
`
`RECEIVE USER REQUEST TO COPY FILE X TO FILE Y .
`
`
`
`ISSUE CLOUD COMMAND TO COPY - FILE X TO FILE Y
`
`TIME
`
`Page 50 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 49 of 60
`
`US 9 , 811 , 662 B2
`
`SNAPSHOT
`
`X . VER 1 . X . VER N - -
`
`FILE X ) . . . O
`
`FILEY [ FILEY )
`
`FIG . 36
`
`D } R2
`
`CLOUD STORAGE SYSTEM 302
`
`CLOUD CONTROLLER 3600
`
`FILESYSTEM VIEW
`PRESENTED TO CLIENT 3604
`
`DIRDIRI
`CLOUDFS
`CLOUDCMD @ Clouds
`ROOT
`CLOUDCMD
`
`DDCP )
`
`
`
`| DDCP - HISTORY CAR RESTORE
`
`. . . DDCP
`
`CLIENT 3602
`
`SNAP VM COPY DB SYNC
`
`Page 51 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 50 of 60
`
`US 9 , 811 , 662 B2
`
`START
`
`USE TWO OR MORE CLOUD CONTROLLERS TO
`COLLECTIVELY MANAGE THE DATA OF A
`DISTRIBUTED FILESYSTEM
`3700
`
`PRESENT A DISTRIBUTED - FILESYSTEM - SPECIFIC
`CAPABILITY TO A CLIENT SYSTEM AS A FILE IN THE
`DISTRIBUTED FILESYSTEM USING A FILE
`ABSTRACTION
`3710
`
`RECEIVE ATA CLOUD CONTROLLER A REQUEST
`FROM THE CLIENT SYSTEM TO ACCESS THE FILE
`3720
`
`EXECUTE A DISTRIBUTED - FILESYSTEM - SPECIFIC
`ACTION IN RESPONSE TO THE FILE ACCESS
`3730
`
`END END
`FIG . 37
`
`Page 52 of 116
`
`Netskope Exhibit 1006
`
`

`

`atent
`
`Nov . 7 , 2017
`
`Sheet 51 of 60
`
`US 9 , 811 , 662 B2
`
`CLOUD STORAGE SYSTEM 302
`
`FIG . 38A
`
`POLICY
`
`FILET . V1 . . . FLET . VN . . . O
`
`SNAPSHOT
`
`POLICY
`
`DIR3
`FILE 1 FILE 20
`
`FILE 1 FILE 2 DIR3
`
`CLOUD CONTROLLER 3600
`
`FILESYSTEM VIEW
`PRESENTED TO CLIENT 3604
`
`CLIENT 3602
`
`ROOT
`
`CLOUDFS
`CLOUDCMD
`
`Page 53 of 116
`
`Netskope Exhibit 1006
`
`

`

`U . S . Patent
`
`Nov . 7 , 2017
`
`Sheet 52 of 60
`
`US 9 , 811 , 662 B2
`
`} O FINO LATEST TO
`
`FIND & ATTEMPT TO
`ACCESS FILE
`
`DETERMINE THAT
`FILE IS ARCHIVED
`
`> cd / cloudfs / fs / dir
`> Is
`file1 file2 dir3
`
`> cat file1
`cat : file1 : file is archived and cannot be read
`> cd . policy
`> is
`file1 file2 dir3
`> cat file1
`# Archived file - not local
`# Current restore delay : 2 days
`
`> echo " Restore " > file 1
`> cat file1
`# Restoring file :
`# restore will complete in 42 hours
`# file data will be unarchived for 14 days
`
`DETERMINE
`RECOVERY
`OPTIONS
`
`AND CHECK
`RESTORE STATUS
`
`! ! ! ! INITIATE RESTORE
`
`< after 42 hours >
`> cat file1
`# Restored file
`# restore complete
`# file data will remain unarchived for 14 days
`
`> cd . .
`> is
`file 1 file2 dir3
`> cat file1
`< receive file data >
`
`FIG . 38B
`
`CONFIRM RECOVERY
`
`ACCESS RES